Thanks @WeatherGod

It might be best to merge the fitting parameters-related features from the __spi__.py script into __main__.py so that we can have everything together in one place. That script was a rush job to facilitate faster processing for a scientist's paper but the new features haven't yet been merged into the __main__.py script. The process should be mostly the same for SPEI.

Maybe an easier intermediate step is to just extend the __spi__.py script so that it accommodates both SPI and SPEI, and once that's working well we then focus on rolling everything from there into __main__.py?

In any event, I totally appreciate your help/involvement, thanks in advance for your work on this.

I got into the nitty-gritty yesterday, and I should have a proof-of-concept by the end of the day (famous last words!). From a CLI perspective, my thinking is that supplying --netcdf_pet should be enough to signal that the user wants SPEI instead of SPI.

Plumbing this through __spi__.py, I am starting to appreciate the difficulty in designing this software. The tons of duplicate code is coming about because the desired combination of operations aren't easily composable with the current design. distributions * scales * timeseries vs. divisions * climate indices = a whole lot of duplication (if-statements on outer loops) or lots of highly inefficient code (if-statements on inner loops). I'll have to think hard on this to see if I can come up with solutions. Naively, I would think passing around functions into the inner loops might work, but I don't know how well that works with numba. The situation might very well be that numba-jitting would actually figure out quickly that ifs in the inner loops aren't changing and optimize them out after the first pass...

But, first things first. Let me get a prototype working for the spi script. There are a number of places where I will point out some possible questionable code or possibly incomplete implementations. I am not sure how ready some of the features are for merging into __main__.py (re: questions about composability and design of the operations).

I'd also use this functionality. But in the meantime could one simply use the current spi script but subtract PET from the precip data manually, before running spi? Essentially the script will think it's calculating SPI but the precip being used would really be precip-PET. Or am I missing something?

Good question. As it stands right now, that idea wouldn't work because negative input values are clipped to 0 just before the sliding sums scaling is performed. Negative PET values (net water loss) is very much possible and would mess up any SPEI calculations.

Right I see, I think you're referring to the several lines from here (there's clipping then an offset to ensure positive numbers). But this could be done manually pretty easily with a few lines of CDO. The way I see it the steps the user would have to take are:

1. clip negative precip to zero
2. P_minus_PET = Precip - PET
3. P_minus_PET = P_minus_PET + 1000 # this is the offset seen at the above code to ensure positive
4. feed the above P_minus_PET into the SPI function. (SPI will still check for negative values but won't find any because of the above steps).

Anyway I'll try check this today.

I have some apprehensions about the adding 1000.0 trick that is currently in the code. I haven't looked to see how that impacts the distribution fits, especially gamma. I think Pearson III would be immune, more or less, given that it has a "loc" parameter. It also isn't a guarantee of positive definiteness. It is highly unlikely to be negative, but not guaranteed.

As a stopgap solution, your idea has merit. But I'd be very careful with any analyses.

Ok I've tested the above steps and can confirm it produces virtually identical results. Image of a time series of the difference in SPEI between the two methods, from one grid cell in my domain:

So in terms of modifying code to save parameter values it might just be a slight change to where data gets treated. Not sure if that helps @monocongo.

RE the +1000 trick, that's an interesting thought. I've noticed my SPEI data for Pearson comes out with obvious spatial discontinuities and is clearly wrong. e.g. see around the middle of the below image (nothing like this happens with Gamma or for either Gamma/Pearson for SPI):

Not sure what causes this, except that I'm using a 20 year base period but even still I'm surprised. If you've seen anything like this would be good to know.

@heroldn There is nothing obviously wrong with your approach for SPEI outlined above. Basically, the SPI is just fitting the data to gamma and PearsonIII, and SPEI just subtracts PET first.

Your observation about bad Pearson results for SPEI as shown in the above map isn't something I've seen before. Thanks for bringing this up. Maybe this is a new/separate issue?

Thanks @monocongo, and yes the pearson map is a separate thing, I shouldn't have polluted this issue. And I'm guessing it might just be poor fits of pearson to my short-ish time series, in which case it's not even a software issue and just how the statistics play out for my data.