Cross validation strategy as 'env' partitioning explaination

[FannyThiery]

Hi all !

I am currently using Biomod2 for whale spatial distribution and I saw that using the 'env' strategy for cross validation improved the ROC and TSS scores for calibration and testing. I was wandering how does 'env' strategy actually partitions the data. Is it geographically (like following the lat and long coordinates to avoid spatial biais, but I think that corresponds more to the 'block' or 'stratified' strategies) or following the range of each variable (to make sure that calib and validation processes cover the range unit of each variable to make sure to train and validate on the whole range) ? I just want to really understand what is behind this strategy to make sure I am correctly using it.

Thank you ! Cheers

[MayaGueguen]

Hello Fanny 👋

It is indeed a good question as the method is not the best described in our documentation 🙈

As you said, the env strategy for cross-validation selection is indeed linked to the range of each environmental variable.
Let's see that with our example data 👀

library(biomod2)
library(terra)

## Load real data ---------------------------------------------------------
data(DataSpecies)
myResp <- as.numeric(DataSpecies[, 'GuloGulo'])

data(bioclim_current)
myExpl <- rast(bioclim_current)

myCoord <- DataSpecies[, c('X_WGS84', 'Y_WGS84')]

# Format Data with true absences
myBiomodData <- BIOMOD_FormatingData(resp.var = myResp,
                                     expl.var = myExpl,
                                     resp.xy = myCoord,
                                     resp.name = 'GuloGulo')

We have 5 variables, and let's focus on bio12 which is annual precipitation :

env <- "bio12"
k <- 3
balance <- which( [email protected] == 1 ) ## presences

tmp.env <- [email protected][balance, env]
bands <- quantile(tmp.env, probs = seq(0, 100, 100 / k) / 100)

I asked for k = 3 partitions, so the function is calculating the quantiles over the range of values of bio12 BUT only considering presence points (as I asked to balance over presences).

> bands
        0%  33.33333%  66.66667%       100% 
  62.21429  299.80554  484.58337 2798.94800 

And then I'm building k datasets, each time taking ALL points that are OUTSIDE the considered range :

full.env <- [email protected][ , env]
calib.env <- matrix(NA, nrow = length(full.env), ncol = k)
for (i in 1:k) {
    calib.env[, i] <- (full.env <= bands[i] | full.env > bands[i + 1])
}

Meaning that, for example, 1st partition will take as calibration all points whose annual precipitation are below 62.21429 and above 299.80554, and keeping the points whose values are within this range for validation.

And this is done for each environmental variables, so in this example, as I have 5 variables, and asked for k = 3, I will end up with 5 * 3 = 15 calibration / validation datasets.

➡️ Is it clearer that way ? Do not hesitate if some things are still not clear, or if I missed a point !

Maya

[FannyThiery]

Hi Maya !

Thank you for your quick answer !

It is indeed a bit more clear. In my case, I chose k= 4, what would be the partitions based on and does the partition of the validation change (as it is the principle of the cross validation strategy, to be able to train and test on different partition of the dataset).

What is the point of doing the partitioning this way ? Why choosing to train on 'the extremes' and validate on the more average data ? I just would like to know the pro and cons of using this method ;)
Why not trying to have the same mean (or median) in each partition instead ?

Thank you again for the details !
Fanny

[MayaGueguen]

• I you choose k = 4, it means that each of your variable will be divided into 4 parts (so here limits between each will correspond to 1st quartile, median and 3rd quartile).
• As you balanced over presences, it means that within each partition, you will have 1/4 of your presences kept for validation, and the rest used for calibration. So it is not training on the extremes. It is similar to a k-fold, but splitted over the range of a variable, rather than a number of points. In each validation set, you will have the same number of presences, but depending whether the presences are uniformly distributed over the range of the variable or not, the range of values for the concerned variable within each validation set might be different in amplitude.
• And for each variable, partitions will indeed be the same, even if you rerun the selection. There is no random part associated to it, unless you change the number k of partitions.
• You will obtain in total 4 * your_number_of_variables CV datasets.

ℹ️ Note that it exists a lot of methods to build cross-validation datasets :

• some more classical like split-in-2, k-fold, leave-on-out, etc with a random aspect
• some "fixed", like spatial block or here "environmental" blocks

➡️ The idea for the spatial block strategy is to try and reduce bias due to spatial autocorrelation (closer data points having similar values).
It's the same idea for_environmental block_ strategy, that might help with data clustered in terms of environmental values (data points similar in terms of precipitation conditions having similar other values). It can also be used to test for model overfitting and to assess transferability in environmental space.

What about now ? 🙂
Maya

[FannyThiery]

Regarding the first point, the way the dataset is splitted, when you say: 1st quartile, median and 3rd quartile, it is of the number of observations, right ?
The dataset is equally splitted in k partitions in terms of number of observations. Am I correct ? And then this will lead to the split of each variable that correspond to the observation in each partition.

I think it is clearer and clearer !

I guess that there is no rule for choosing one strategy or another. The best is to try couple and see which one captures the most information/variability and so will correspond the best to the dataset, right ?

Thank you very much for your answers and for the amazing job you do with this very useful package !
Fanny

[FannyThiery]

Ok I think I understood... The dataset is splitted following the distribution of the presences over each environmental variables right ?
It equally divides the number of presences of the dataset but over the range of the environmental variables. If the studied species 'prefers' a narrow high range of temperature for example but for some reason I have data for more extreme low values I will have my 4 partitions splitted for example like 0 - 5, 5 - 10, 10 - 15 and 15 -16 because I will have the same amount of presences between 15 and 16° and in the other partitions.

Thanks again,
Fanny

[MayaGueguen]

Yes, you got it !

For each variable, I am looking at all the points that are presences, retrieving the values of the concerned variable for those points, and computing my requested quantiles over those values. This give me the thresholds that delimit how I will select, among all points, the different partitions, each time keeping one part for validation, and the remaining for calibration.

Your graphic got it right, for balance = "presences" and k = 4 🥇

🙂
Maya

[FannyThiery]