Regression_estimator.qmd

---
title: "Applications of Small Area Estimation Methods in Forest Inventories and Modeling. Center for Intensive Planted-forest Silviculture"
author: "Francisco Mauro, Temesgen Hailemariam &  Bryce Frank"
format: pdf
editor: visual
---

# 1. Introduction

```{r}
library("sae")
library("nlme")
library("sf")
library("terra")
library("raster")
library("mapview")
library("dplyr")
library("tidyr")
library("ggplot2")
library("gridExtra")
library("leaps")
library("caret")
library("car")
```

This markdown document covers the principal concepts of regression estimators, unit-level models and area-level models for small area estimation.

# 2. Regression estimator

We are going to compare regression estimators with the traditional sample mean for an artificial population that we will use throughout the examples. In this example we have the value of volume for all grid units in the forest along with an auxiliary variable called p95. **THIS NEVER HAPPENS IN REAL LIFE** but will help us understand regression estimators. We will mimic simple random sampling and compute the normal expansion estimator (sample mean) to a regression estimator.

Our population mimics a square tract of forest of 160 acres with four stands 40 of acres each one. Let's take a look at the data.

```{r}
library(ggplot2)
example1 <- read.csv("example1.csv")

#PLACEHOLDER
# paste here

example1$stand <- factor(example1$stand)
ggplot(example1,aes(x=x,y=y,fill=stand)) + geom_tile()
ggplot(example1,aes(x=x,y=y,fill=volume)) + geom_tile()
ggplot(example1,aes(x=x,y=y,fill=p95)) + geom_tile()
plot(example1$p95,example1$volume,col=example1$stand)
```

We will start estimating the average volume in the forest and in the mean volume in each stand doing simple random sampling without replacement. Having the volume of all grid units allows us to obtain the true average volume in the forest.

```{r}
true_mean_forest <- mean(example1$volume)
true_mean_stand1 <- mean(example1$volume[example1$stand==1])
true_mean_stand2 <- mean(example1$volume[example1$stand==2])
true_mean_stand3 <- mean(example1$volume[example1$stand==3])
true_mean_stand4 <- mean(example1$volume[example1$stand==4])
print("True mean")
true_mean_forest
print("True mean stand 1")
true_mean_stand1
print("True mean stand 2")
true_mean_stand2
print("True mean stand 3")
true_mean_stand3
print("True mean stand 4")
true_mean_stand4
range_volume <- range(example1$volume)

```

## Sample mean with simple random sampling

Let's simulate our simple random sampling. We will repeat the simple random sampling 100 with sample sizes of 16, 64, 96, 128, 160, 320 and 640 and plot our estimates to visualize the variance of doing simple random sampling. We will store the result of each iteration in a data.frame with our estimate, the sample size and the iteration.

```{r}
repeats <- 100
sample_sizes <- c(16, 64, 96, 128, 160, 320, 640 )
result <- data.frame(rep=c(),sample_size=c(),
                      estimated_mean_forest=c(),
                      estimated_mean_stand1 = c(),
                      estimated_mean_stand2 = c(),
                      estimated_mean_stand3 = c(),
                      estimated_mean_stand4=c())
cont <- 1
for(i in 1:repeats){
  for(j in sample_sizes){
    # get a sample of the indices of the data
    my_sample<- sample(dim(example1)[1],j,replace = FALSE)
    # subset the forest
    sampled_data <- example1[my_sample,]
    # store the rep and sample size
    result[cont,"rep"]<-i
    result[cont,"sample_size"]<-j
    # calculate the sample means for the forest and each stand
    result[cont,"estimated_mean_forest"]<-mean(sampled_data$volume)
    result[cont,"estimated_mean_stand1"]<-mean(sampled_data$volume[sampled_data$stand==1])
    result[cont,"estimated_mean_stand2"]<-mean(sampled_data$volume[sampled_data$stand==2])
    result[cont,"estimated_mean_stand3"]<-mean(sampled_data$volume[sampled_data$stand==3])
    result[cont,"estimated_mean_stand4"]<-mean(sampled_data$volume[sampled_data$stand==4])
    # pass to next iteration
    cont <- cont+1
  }
}
# Store the pre-calculated true values
result$true_mean_forest<-true_mean_forest
result$true_mean_stand1 <- true_mean_stand1
result$true_mean_stand2 <- true_mean_stand2 
result$true_mean_stand3 <- true_mean_stand3 
result$true_mean_stand4 <- true_mean_stand4

```

Let's now plot our results as a function of the sample size. We will simulate a precision requirement of +- 10% in our estimates piloting discontinuous lines around the true value

```{r}

plot(result$sample_size,result$estimated_mean_forest,ylim=range_volume,
      main="Estimated means forest vs sample size")
abline(true_mean_forest,0,col="blue")
par(mfrow=c(2,2))
plot(result$sample_size,result$estimated_mean_stand1,ylim=range_volume,
      main="Estimated means stand 1 vs sample size")
abline(true_mean_stand1,0,col="red")
plot(result$sample_size,result$estimated_mean_stand2,ylim=range_volume,
      main="Estimated means stand 2 vs sample size")
abline(true_mean_stand2,0,col="green")
plot(result$sample_size,result$estimated_mean_stand3,ylim=range_volume,
      main="Estimated means stand 3 vs sample size")
abline(true_mean_stand3,0,col="purple")
plot(result$sample_size,result$estimated_mean_stand4,ylim=range_volume,
      main="Estimated means stand 4 vs sample size")
abline(true_mean_stand4,0,col="orange")

par(mfrow=c(1,2))
plot(result$sample_size,result$estimated_mean_forest,ylim=range_volume,
     main="Estimated means forest vs sample size")
abline(true_mean_forest,0,col="blue")
abline(true_mean_forest+0.1*true_mean_stand4,0,col="blue",lty=2)
abline(true_mean_forest-0.1*true_mean_stand4,0,col="blue",lty=2)
plot(result$sample_size,result$estimated_mean_stand4,ylim=range_volume,
      main="Estimated means stand 4 vs sample size")
abline(true_mean_stand4,0,col="orange")
abline(true_mean_stand4+0.1*true_mean_stand4,0,col="orange",lty=2)
abline(true_mean_stand4-0.1*true_mean_stand4,0,col="orange",lty=2)

```

Inspecting the generated figures we can observe the variance of our estimates as a function of sample size and draw some interesting conclusions.

1.  The variability of our estimates over repeated samples decreases with a pattern that resembles the graph of $\sqrt{\frac{1}{n}}$

2.  As we know the sample mean is unbiased and, on average, tend to coincide with the true means for the forest and the stands.

3.  When we compare the figure for the total forest and the figure for stand 4, we see that at some point, the sample size for stand 4 becomes so small that the variability is very large.

Now we are going to repeat the same process but instead of using the sample mean we will use the auxiliary information to calculate the regression estimator using p95 as auxiliary variable (X).

```{r}
repeats <- 100
sample_sizes <- c(16, 64, 96, 128, 160, 320, 640 )
result_reg <- data.frame(rep=c(),sample_size=c(),
                      estimated_mean_forest=c(),
                      estimated_mean_stand1 = c(),
                      estimated_mean_stand2 = c(),
                      estimated_mean_stand3 = c(),
                      estimated_mean_stand4=c())
cont <- 1
for(i in 1:repeats){
  for(j in sample_sizes){
    # get a sample of the indices of the data
    my_sample_reg<- sample(dim(example1)[1],j,replace = FALSE)
    # subset the forest and obtain the sample fit using OLS (lm function)
    sampled_data_reg <- example1[my_sample_reg,]
    model_forest <- lm(volume~p95,data = sampled_data_reg)
    # calculate the regression estimator
    media_reg_forest <- mean(predict(model_forest,example1)) + mean(model_forest$residuals)
    
    # now we will create subsamples for stands, obtain subsample fits for the stand and apply 
    # the regression estimator
    stand1 <- sampled_data_reg[sampled_data_reg$stand==1,]
    media_reg_stand1 <- try({
      model_stand1 <- lm(volume~p95, data=stand1)
      mean(predict(model_stand1,example1[example1$stand==1,]))+ mean(model_stand1$residuals)
    })
    if(inherits(media_reg_stand1,"try-error")){
      media_reg_stand1<-NA
    }
    # repeat for stand 2
    stand2 <- sampled_data_reg[sampled_data_reg$stand==2,]
    media_reg_stand2 <- try({
      model_stand2 <- lm(volume~p95, data=stand2)
      mean(predict(model_stand2,example1[example1$stand==2,])) + mean(model_stand2$residuals)
    })
    # bypass very low stand sample size
    if(inherits(media_reg_stand2,"try-error")){
      media_reg_stand2<-NA
    }
    # repeat for stand 3
    stand3 <- sampled_data_reg[sampled_data_reg$stand==3,]
    media_reg_stand3 <- try({
      model_stand3 <- lm(volume~p95, data=stand3)
      mean(predict(model_stand3,example1[example1$stand==3,])) + mean(model_stand3$residuals)
    })
    # bypass very low stand sample size
    if(inherits(media_reg_stand3,"try-error")){
      media_reg_stand3<-NA
    }
    
    # repeat for stand 4
    stand4 <- sampled_data_reg[sampled_data_reg$stand==4,]
    media_reg_stand4 <- try({
      model_stand4 <- lm(volume~p95, data=stand4)
      mean(predict(model_stand4,example1[example1$stand==4,])) + mean(model_stand4$residuals)
    })
    # bypass very low stand sample size
    if(inherits(media_reg_stand4,"try-error")){
      media_reg_stand4<-NA
    }
    
    
    
    stand2 <- sampled_data_reg[sampled_data_reg$stand==2,]
    stand3 <- sampled_data_reg[sampled_data_reg$stand==3,]
    stand4 <- sampled_data_reg[sampled_data_reg$stand==4,]
    # store the rep and sample size
    result_reg[cont,"rep"]<-i
    result_reg[cont,"sample_size"]<-j
    # calculate the sample means for the forest and each stand
    result_reg[cont,"estimated_mean_forest"]<-media_reg_forest
    result_reg[cont,"estimated_mean_stand1"]<-media_reg_stand1
    result_reg[cont,"estimated_mean_stand2"]<-media_reg_stand2
    result_reg[cont,"estimated_mean_stand3"]<-media_reg_stand3
    result_reg[cont,"estimated_mean_stand4"]<-media_reg_stand4
    # pass to next iteration
    cont <- cont+1
  }
}
# Store the pre-calculated true values
result_reg$true_mean_forest<-true_mean_forest
result_reg$true_mean_stand1 <- true_mean_stand1
result_reg$true_mean_stand2 <- true_mean_stand2 
result_reg$true_mean_stand3 <- true_mean_stand3 
result_reg$true_mean_stand4 <- true_mean_stand4
```

No let's plot the results

```{r}
plot(result_reg$sample_size,result_reg$estimated_mean_forest,ylim=range_volume,
      main="Estimated means forest vs sample size")
abline(true_mean_forest,0,col="blue")
par(mfrow=c(2,2))
plot(result_reg$sample_size,result_reg$estimated_mean_stand1,ylim=range_volume,
      main="Estimated means stand 1 vs sample size")
abline(true_mean_stand1,0,col="red")
plot(result_reg$sample_size,result_reg$estimated_mean_stand2,ylim=range_volume,
      main="Estimated means stand 2 vs sample size")
abline(true_mean_stand2,0,col="green")
plot(result_reg$sample_size,result_reg$estimated_mean_stand3,ylim=range_volume,
      main="Estimated means stand 3 vs sample size")
abline(true_mean_stand3,0,col="purple")
plot(result_reg$sample_size,result_reg$estimated_mean_stand4,ylim=range_volume,
      main="Estimated means stand 4 vs sample size")
abline(true_mean_stand4,0,col="orange")

par(mfrow=c(1,2))
plot(result_reg$sample_size,result_reg$estimated_mean_forest,ylim=range_volume,
     main="Estimated means forest vs sample size")
abline(true_mean_forest,0,col="blue")
abline(true_mean_forest+0.1*true_mean_stand4,0,col="blue",lty=2)
abline(true_mean_forest-0.1*true_mean_stand4,0,col="blue",lty=2)
plot(result_reg$sample_size,result_reg$estimated_mean_stand4,ylim=range_volume,
      main="Estimated means stand 4 vs sample size")
abline(true_mean_stand4,0,col="orange")
abline(true_mean_stand4+0.1*true_mean_stand4,0,col="orange",lty=2)
abline(true_mean_stand4-0.1*true_mean_stand4,0,col="orange",lty=2)
```

If we observe the plots for the regression estimator we can see:

-   The pattern with $\sqrt{\frac{1}{n}}$ is also there, but this time the variability is smaller.

-   For all sample sizes, the estimates over repeated samples concentrate around the true values.

-   As before, the estimates for the entire forest have much lower variability than estimates for single stands.

-   You probably obtained some warnings in the process because some stands were never sampled or had very small sampled sizes.

## Regression estimator with simple random sampling

Let's now compare use the regression estimator. We will simulate a precision requirement of +- 10% in our estimates plotting discontinuous lines around the true value

```{r}
par(mfrow=c(2,2))
plot(result_reg$sample_size,result_reg$estimated_mean_forest,ylim=range_volume,
     main="Regression estimator")
abline(true_mean_forest,0,col="blue")
abline(true_mean_forest+0.1*true_mean_forest,0,col="blue",lty=2)
abline(true_mean_forest-0.1*true_mean_forest,0,col="blue",lty=2)
plot(result$sample_size,result$estimated_mean_forest,ylim=range_volume,
     main="Sample mean")
abline(true_mean_forest,0,col="blue")
abline(true_mean_forest+0.1*true_mean_forest,0,col="blue",lty=2)
abline(true_mean_forest-0.1*true_mean_forest,0,col="blue",lty=2)


plot(result_reg$sample_size,result_reg$estimated_mean_stand4,ylim=range_volume,
      main="Regression estimator")
abline(true_mean_stand4,0,col="orange")
abline(true_mean_stand4+0.1*true_mean_stand4,0,col="orange",lty=2)
abline(true_mean_stand4-0.1*true_mean_stand4,0,col="orange",lty=2)
plot(result_reg$sample_size,result_reg$estimated_mean_stand4,ylim=range_volume,
      main="Sample mean")
abline(true_mean_stand4,0,col="orange")
abline(true_mean_stand4+0.1*true_mean_stand4,0,col="orange",lty=2)
abline(true_mean_stand4-0.1*true_mean_stand4,0,col="orange",lty=2)
```

When comparing the sample mean to the regression estimator, we can observe that:

-   For the entire forest the regression estimator does much better for all sample sizes. This is because the relationship between volume and p95 can be approximated well by a linear function. The relationship, however, is not linear, it was created with the code spinet below, but this method does not rely on assuming that a linear relationship exists.

-   For both methods variability for stands was larger and for small sample sizes variances were large.

[**This is the small area estimation problem. We have reach to a point where even using auxiliary information with a large correlation with our response variable does not allow us to obtain reliable estimates.**]{.underline}

```{r}
id <- 1:1600
x <- rep(0:39,each=40)
y <- rep(0:39,times=40)
stand <- (1+x%/%20) +2*(y%/%20)
p95 <- rnorm(1600,50,10+5*stand)
volume <- rnorm(1600,p95+0.05*p95^2+100*stand+50,60)
example1 <- data.frame(id=id,x=x,y=y,stand=stand,volume=volume, p95=p95)
```

Take the code snippet above and replace the #PLACEHOLDER comment in the first cell of code with a data pattern generated by you for volume. You can make the data have a more or less linear relationship. Then re-run the code and see what happens.

## **Important notes**

All our assessment were based on repeating the sampling process. That sampling process implemented the design that we chose (simple random sampling). There are some things to consider for general sampling designs:

-   For simple random sampling and systematic designs we use ordinary least square to estimate the line of best fit. For other general sampling designs that does not necessarily hold. Read Särndal et al., (2003) section 6.5 for a more general formulation of regression estimators for general sampling designs.

-   We have used an example in which we know the entire population. This is appropriate to see how regression estimators work. In real applications we will only have the variable of interest for the sample. In those cases we will have to use formulas to estimate the variance of our regression estimator. Särndal et al., (2003) also provides those formulas and they will provide estimates of the spread of the points that we obtain for each sample size.

# 3. Model-based synthetic estimator

To start with the regression estimator we are going to use the data in example2.csv. It is the data we used in example one but the volume column has been removed so we can simulate different models.

```{r}
example2 <- read.csv("example2.csv")
```

Lets start assuming some model form.

$$
Vol_i\sim N(25+6*P95,130^2) 
$$

With $Cov(Vol_i,Vol_j)=0$ for all $i$ and $j$

Element 200 in the population has a value of P95 equal to 43.06279, according to this value the volume for this unit is distributed normally with mean 25+6\*43.06279 = 283.3767, and variance 200.

$$
Vol_{200}\sim N(283.3767,130^2)
$$ We can calculate the distribution of the mean volume.

$$
\bar{Vol} \sim N(\frac{1}{1600}\sum_{i=1}^{1600}(25+6P95_i),\frac{1}{1600}130^2)= N(323.8917,3.25^2)
$$

According to the model the distribution of the mean volume in the forest is:

```{r}
mean_vol_dist <- mean(25+6*example2$p95)
mean_vol_dist
plot(mean_vol_dist+c(-100:100)/10,
     dnorm(mean_vol_dist+c(-100:100)/10,
           mean=mean_vol_dist,
           130/sqrt(1600)),
     type="l",ylab="Probability density",xlab="Mean Volume")

```

Lets compare the distribution of the mean to the distribution of given unit centered also at 323.8917.

```{r}
par(mfrow=c(1,2))
plot(x=mean_vol_dist+c(-10000:10000)/10,xlim=c(0,600),
     y=dnorm(mean_vol_dist+c(-10000:10000)/10,
     mean=mean_vol_dist, 130),col="red",
     type="l",ylab="Probability density",xlab="Mean Volume",
     main="Single unit")
plot(x=mean_vol_dist+c(-10000:10000)/10,xlim=c(0,600),
     y=dnorm(mean_vol_dist+c(-10000:10000)/10,
     mean=mean_vol_dist, 130/sqrt(1600)),col="blue",
     type="l",ylab="Probability density",xlab="Mean Volume",
     main="Mean of all units")

```

We can see that assuming independence is a very strong assumption. With independence departures there is a large compensation. Some units deviate above the mean and some units deviate below the mean causing the variance of the mean to be very small. We will see that this assumption is not realistic in many settings. For example, we can expect volume for units within a give stand to show some correlation. Stands are differentiated units, someone draw the stand boundaries for a reason, and it is reasonable to expect some within stand correlation.

## A more realistic example (still assuming independence)

This is a more realistic example. We have a similar population, in this case we have 16 smaller stands. And we have sampled 100 units in the field so we have observed volume for those 100 units. For the rest of the population we do not know volume. For all populations we have the values of p95.

```{r}
example3 <- read.csv("example3.csv")
example3$stand <- factor(example3$stand)
ggplot(example3,aes(x=x,y=y,fill=stand)) + geom_tile()
plot(example3$p95,example3$volume,pch=20)
```

We are going to make the hypothesis that there is a linear relationship between volume and p95.

$$
Vol_i\sim N(a+b*P95,\sigma^2)\:with\:Cov(Vol_i,Vol_j) =0\:for\:all\:i\:and\:j
$$

We are going to fit the model using the sampled units, so we will divide the data into sampled and unsampled parts.

```{r}
sampled <- example3[!is.na(example3$volume),]
unsampled <- example3[!is.na(example3$volume),]
```

Now using the sampled part we are going to fit a linear

```{r}
model <- lm(volume~p95,data=sampled)
summary(model)
sigma_e <- summary(model)$sigma
sigma_e
plot(model)
```

We have some outliers but we will accept this model to see how we obtain estimates and associated uncertainties with the fitted model. The point estimate is obtained predicting for all elements of the population using the model fitted with the observed sample.

```{r}
# The prediction for the mean in the forest is obtained using the coefficients of the model fitted with the sample
predicted<-mean(predict(model,example3))
predicted
```

To assess the uncertainty of our estimate we consider alternative values, compatible with the model, for the sampled and unsampled parts. We will work with a sample of 50 units which implies \~3 plots per stand. We will use parametric bootstratping and repeat the process of fitting the model and computing predictions for all population units 1000 times There are some differences with the regression estimator:

1.  In a model-based setting, the randomness comes from the model (data generating process). We are not going to randomize the sample. We have sampled 50 units and those units will remain fixed.

2.  Randomness come from the model , so we will generate alternative values for the sampled and unsampled data (compatible with the model) and repeat the process in the code chunk above several times.

3.  The mean volume for the forest and the stand are also random variables themselves. They are not fixed quantities that we can take as a reference, therefore, we will generate the true value for the forest and each stand in every iteration using the model. As these quantities are random, we cannot use them as a reference. They change in every iteration. Thus, we will measure the differences between the mean values for the forest and each stand and the predicted values obtain in each iteration.

```{r}
repeats <- 1000
differences <- data.frame(rep=c(),
                      true_mean_forest=c(),
                      true_mean_stand1 = c(),
                      true_mean_stand2 = c(),
                      true_mean_stand3 = c(),
                      pred_mean_forest=c(),
                      pred_mean_stand1 = c(),
                      pred_mean_stand2 = c(),
                      pred_mean_stand3 = c(),
                      mean_stand4=c(),
                      diff_mean_forest=c(),
                      diff_mean_stand1 = c(),
                      diff_mean_stand2 = c(),
                      diff_mean_stand3 = c(),
                      diff_mean_stand4=c())

for(i in 1:repeats){
  # create a copy of sampled and unsampled
  sampled_i <- sampled
  unsampled_i <- unsampled
  
  # generate values of volume according to the model
  sampled_i$volume <- predict(model,sampled_i) + 
    rnorm(dim(sampled_i)[1],0,sigma_e)
  unsampled_i$volume <- predict(model,unsampled_i) + 
    rnorm(dim(unsampled_i)[1],0,sigma_e)
  
  # merge the data in a single data frame for convenience
  population_i <- rbind(sampled_i,unsampled_i)
  # repeat the process of fitting the model parameters and calculating 
  # the mean for the population using the model coefficients
  model_i <- lm(volume~p95,sampled_i)
  population_i$prediction <- predict(model_i,population_i)
  predicted_i <- mean(population_i$prediction)
  
  
  mean_forest_i <- mean(population_i$volume)
  mean_stand1_i <- mean(population_i[population_i$stand==1,"volume"])
  mean_stand2_i <- mean(population_i[population_i$stand==2,"volume"])
  mean_stand3_i <- mean(population_i[population_i$stand==3,"volume"])
  mean_stand4_i<- mean(population_i[population_i$stand==4,"volume"])
  
  pred_mean_forest_i <- mean(population_i$prediction)
  pred_mean_stand1_i <- mean(population_i[population_i$stand==1,"prediction"])
  pred_mean_stand2_i <- mean(population_i[population_i$stand==2,"prediction"])
  pred_mean_stand3_i <- mean(population_i[population_i$stand==3,"prediction"])
  pred_mean_stand4_i <- mean(population_i[population_i$stand==4,"prediction"])
  
  # store the rep and sample size
  result_reg[cont,"rep"]<-i
  #Store the values of the true mean for the forest and the stands in each iteration
  result_reg[i,"true_mean_forest"]<-mean_forest_i
  result_reg[i,"true_mean_stand1"]<-mean_stand1_i
  result_reg[i,"true_mean_stand2"]<-mean_stand2_i
  result_reg[i,"true_mean_stand3"]<-mean_stand3_i
  result_reg[i,"true_mean_stand4"]<-mean_stand4_i
  
  # Store the predicted values for the mean for the forest (not the estimated mean)
  result_reg[i,"pred_mean_forest"]<-pred_mean_forest_i
  result_reg[i,"pred_mean_stand1"]<-pred_mean_stand1_i
  result_reg[i,"pred_mean_stand2"]<-pred_mean_stand2_i
  result_reg[i,"pred_mean_stand3"]<-pred_mean_stand3_i
  result_reg[i,"pred_mean_stand4"]<-pred_mean_stand4_i
  # calculate the sample means for the forest and each stand
  result_reg[i,"diff_mean_forest"]<-mean_forest_i-pred_mean_forest_i
  result_reg[i,"diff_mean_stand1"]<-mean_stand1_i-pred_mean_stand1_i
  result_reg[i,"diff_mean_stand2"]<-mean_stand2_i-pred_mean_stand2_i
  result_reg[i,"diff_mean_stand3"]<-mean_stand3_i-pred_mean_stand3_i
  result_reg[i,"diff_mean_stand4"]<-mean_stand4_i-pred_mean_stand4_i

}
```

Let's see the how important the differences between our predictions and the mean value in the forest and in the stand can be if the model holds.

```{r}
hist(result_reg$diff_mean_forest)
hist(result_reg$diff_mean_stand1)
hist(result_reg$diff_mean_stand2)
hist(result_reg$diff_mean_stand3)
hist(result_reg$diff_mean_stand4)
```

These results look decent. Now we look at the predicted values for all repetitions.

```{r}
hist(result_reg$pred_mean_forest)
hist(result_reg$pred_mean_stand1)
hist(result_reg$pred_mean_stand2)
hist(result_reg$pred_mean_stand3)
hist(result_reg$pred_mean_stand4)
```

Check the true model for volume and how the data was actually generated; plot observations from different stands with different colors. Pay attention to the magnitude of the quantities in stand_effects.

```{r}
set.seed(1234)
id <- 1:1600
x <- rep(0:39,each=40)
y <- rep(0:39,times=40)


stand <- (1+x%/%10) +4*(y%/%10)
stand_effect <- rnorm(16,0,150)
stand_effect
p95 <- rnorm(1600,180,20)
volume <- rnorm(1600,5+15*p95+stand_effect[stand],50)

example3 <- data.frame(id=id,x=x,y=y,stand=stand,volume=volume, p95=p95)
example3$stand <- factor(example3$stand)
stands_1_to_4 <- example3[example3$stand%in%1:4,]
plot(stands_1_to_4$p95,stands_1_to_4$volume,col=stands_1_to_4$stand,pch=20)

```

Based on the true data generating process, we would expect differences of up to 220 between stands 1 and 2. Why are we getting values that are almost the same for all stands and for the forest? The model has to be failing in some way. Even worse, we have analyzed the uncertainty based on the assumption that the model holds but it seems that the model we are fitting does not hold. Can we trust this model? Are we missing something?

[**We are not modeling stand effects. This type of modeling is sometimes called synthetic, it assumes that a general model holds for all stands, but that has important risks if stands are actually differentiated units, and if someone draw a stand boundary it is probably because something looked different from the surrounding area.**]{.underline}

# 4. Small area Unit-level model

In the previous sections we revised the synthetic estimator and observed that there are situations in which it can be risky. What can we do? For these cases. There are two solutions that are not optimal:

1.  Use regression estimators for each stand. But we have about 3 plots per stand and chances are that we have stands with no plots. In the best case, regression estimators for the stand will have a large variance because of the small sampled size.

2.  We could change the model so it includes a different intercept for each stand.

We will fit a model with a fixed coefficient lifting or pushing down the regression line for each stand.

```{r}
model_fixed_effects <- lm(volume~p95 + factor(stand)-1,data=sampled) 
stand_effect
summary(model_fixed_effects)
```

In this model the intercepts for the stands can take any value (we are not assuming anything about how they are generated) and to estimate them we have only about 3 plots per stand . This will remove biases, but our estimates will have a large variance because the stand coefficients are estimated with a very small number of plots per stand.

## Is there any other solution?

The answer is yes...but that is something that we will cover after the break...

## Packages to estimate with unit-level models

There are several packages that allow us working with unit level models. Below is a description of the main packages that we can use.

### sae package

This package covers the main small area estimation models. It provides functions to fit unit-level models and also area-level models and computes predictions for small areas (i.e., stands) providing metrics of uncertainty (mean square errors) for those estimates. It is the most direct package to implement small area estimators.

[**Pros:**]{.underline}

-   Fits basic unit level model, basic area level model and for area level models with spatial, temporal and spatio-temporal correlations.

-   Provides estimates for the considered models for small areas.

-   Provides uncertainty for the estimates for small areas.

-   It also includes design based estimators.

-   Easy to use.

[**Cons:**]{.underline}

-   Some cases of interest in forestry are not considered (e.g.,estimates for a group of stands or small areas).

-   Provides tabular summaries of the model but does not return the model itself or allows for graphical summaries.

### nlme and lme4

[**Pros:**]{.underline}

This package provides tools to fit fixed-effects and mixed-effects linear and non-linear models.

-   Highly flexible.

-   Complex hierarchies of random-effects.

-   Some graphical summaries

-   Provides a `predict` function which allows generating raster layers.

-   Excellent book documenting the use of the `nlme` package (Pinheiro and Bates, 2000)

-   In addition to modeling random effects, it allows modeling:

    -   Heteroscedaskicity

    -   Error correlations (spatial, temporal or between variables)

[**Cons:**]{.underline}

-   Learning curve

-   Once you obtain the model parameters, you have compute stand level estimates and uncertainty metrics (harder). To do this, the best reference is the Small Area Estimation book from Rao and Molina (2015).

Let's install these packages and some extra packages to continue with our examples.

## Example unit-level

To work with unit-level models we need plots with coordinates that we use to extract their auxiliary information. In this case, we have already extracted two auxiliary variables from lidar data. The lidar data was collected in 2009 and plots were measured in 2010. The auxiliary variables are the 95th percentile of the heights of lidar returns and the standard deviation of those heights.

### Explore the data

We have a raster layer with lidar data and the stand IDs. It is important to ensure that band names in that raster match the names of the data in the sample object.The object plots is an sf object, we create non spatial copy, called plots_df, by removing the geometry (geom) field. Our small areas are the management units in the study area (stands). The field ID_SMA is the identifier of the stand and matches the plots. The field P95 is the 95th percentile, the field SD_H is the standard deviation of lidar heights and ID_SMA is the ID, of the stand in which the plot was collected. On average there are 3-4 plots per stand.

```{r}
population_aux_info <- rast("Example_Unit_Level.tif")
names(population_aux_info) <- c("P95", "SD_H", "ID_SMA") #Ensure consistent names
plots <- st_read("Field_plots.gpkg")
colnames(plots)[5] <- "QMD"
plots_df <- st_drop_geometry(plots)

stands <- st_read("Stands.gpkg")
mapview(raster(population_aux_info[["P95"]]),legend=FALSE) + 
  mapview(raster(population_aux_info[["SD_H"]]), legend=FALSE) + 
  mapview(raster(population_aux_info[["ID_SMA"]]), legend=FALSE) + 
  mapview(stands, legend=FALSE) + mapview(plots)  
```

Stand sample size and average sample size by stand

```{r}
head(plots_df[-c(1:3),c("Plot_ID","ID_SMA","QMD","V","P95","SD_H")])
n_by_stand  <- plots_df |> group_by(ID_SMA) |> summarize(ni=n())
# Add unsampled stands by merging withh the stands objects.
n_by_stand <- merge(st_drop_geometry(stands)[,"ID_SMA" ,drop=FALSE],
                    n_by_stand,all.x=TRUE)
n_by_stand$ni <- ifelse(is.na(n_by_stand$ni),0,n_by_stand$ni)
head(n_by_stand)
```

```{r}
mean(n_by_stand$ni)
```

We will start creating models for the quadratic mean diameter in the stands (QMD). For that we will start with a small exploratory analysis. P95 and QMD have a strong linear relationship.

```{r}
pairs(data.frame(plots)[, c("QMD", "V","P95", "SD_H")])

```

### Fitting a unit level with nlme

We will start fitting the unit level model using the lme (linear mixed-effects) function from nlme. To specify a random intercept for the stands, we indicate that we want to have a random intercept (in R formula lingo, 1 represents the intercept, the vertical bar is the conditioning symbol and ID_SMA is the column that stores stand IDs), to do so we use random = \~1\|ID_SMA.

```{r}
model <- lme(QMD ~ P95, random = ~ 1 | ID_SMA, data = plots)
model_lm <- lm(QMD ~ P95, data=plots)
anova(model,model_lm)
```

A quick look at the model summary

```{r}
summary(model)
```

and to the residuals

```{r}
plot(model)
```

```{r}
par(mfrow = c(1, 2))

qqnorm(residuals(model, level = 1), main = "Normal Q-Q plot residuals")
qqline(residuals(model, level = 1))

qqnorm(model$coefficients$random$ID_SMA, main = expression("Normal Q-Q plot" ~ hat(v)[i]))
qqline(model$coefficients$random$ID_SMA)
```

The lme model can be used to obtain pixel level predictions.

```{r}
preds <- predict(population_aux_info, model, level=1)
plot(preds)
```

We could aggregate these predictions using zonal stats, however, this approach does not let us get unbiased estimates of the mse. To get that it would be necessary to use the model coefficients and implement the mse estimators ourselves.

### Using the sae package

To get point estimates of QMD for stands and their associated uncertainty we can use the `eblupBHF` and `pbmseBHF` functions in the sae package. These functions the following inputs

1.  The model formula for the fixed effects,

2.  The dom argument, the field that stores the stand identifier,

3.  The meanxpop argument, a data frame with the average of the predictors within stands. The first field is the stand identifier

4.  The popnsize argument, a data frame with the stand ids in the first column and the population sizes in the second column

`eblupBHF` provides estimates for stands and the fitted model using the 4 inputs listed above. `pbmseBHF` estimates the mse using parametric bootstrap, it requires an additional input

5.  B, which is the number replicates for the parametric bootstrap.

**IMPORTANT** To get 3) and 4) we need access to the auxiliary information for the entire population. That information is in the raster file "Examples_Unit_Level.tif". The first band contains P95, the second contains SD_H and the third the stand IDs. we rename the bands so the names match the plots data frame.

To get 3) we will use the zonal function of the `terra` package with the mean function or zonal statistics (ArcGIS or QGIS).

```{r}
X_mean <- zonal(population_aux_info[[c("P95", "SD_H")]],
  population_aux_info[["ID_SMA"]],
  fun = mean, na.rm = TRUE
)
X_mean <- merge(X_mean,n_by_stand,by="ID_SMA")
```

There are some unsampled stands, we will only keep the ones that are sampled

```{r}
head(X_mean)
```

To get 4) we will use the zonal function of the `terra` package with the `length` function

```{r}
Popn <- zonal(population_aux_info[["P95"]],
  population_aux_info[["ID_SMA"]],
  fun = length
)
colnames(Popn)<-c("ID_SMA","N")

X_mean <- merge(Popn,X_mean,by="ID_SMA")
print(X_mean)
```

Once we have 3) and 4) we can use the `eblupBHF` in the `sae` package, attach the plots object so `eblupBHF` will find the information it needs. `eblupBHF` obtain point estimates using the basic unit level model .

```{r}
mean_xpop <- X_mean[,c("ID_SMA","P95")]
popn_size <- X_mean[,c("ID_SMA","N")]
doms <- plots_df$ID_SMA
# we have to add unsampled stands in the plots data.frame
plots_df <- merge(plots_df,X_mean[ , "ID_SMA",drop=FALSE],all.y=TRUE)
# and store all domains in a vector
doms <- plots_df$ID_SMA
result <- eblupBHF(QMD ~ P95, dom = ID_SMA, selectdom=doms,
                   meanxpop = mean_xpop,
                   popnsize = popn_size, data=plots_df)
```

Model fit and point estimates for stands are in the elements fit and eblup (estimates) of the element est of result. They are stored as a list and a data.frame respectively. Model fit:

```{r}
print(result$eblup$eblup)
result$fit$summary
```

The column eblup stores the stand level estimates.

```{r}
head(result$est$eblup)
```

Estimated mean square errors are stored as a data.frame in the mse element of the result. Both, estimates and mse can be merged. We can create a column with the rmses to compute coefficients of variation. Once we merge estimates and mses\\rmses we can get the CVs and relative errors. The field ID_SMA is renamed "domain".

```{r}
eblups <-result$eblup
```

### Uncertainty estimation

The package sae provides estimates of uncertainty using the `pbmse*` functions. For the unit-level model we use the function `pbmseBHF.` In these functions pb stands for [**parametric bootstrap**]{.underline} and in `pbmseBHF` BHF stands for **Battese, Harter and Fuller**, the statisticians that developed these estimators. This technique is similar to the one used when introducing model based estimators but it entails many complexities because now we are using a model with random effects.

```{r}
result <- pbmseBHF(QMD ~ P95, dom = ID_SMA, selectdom=doms,
                   meanxpop = mean_xpop,
                   popnsize = popn_size, B = 100,data=plots_df)
```

To generate outputs that we can share with gis users we are going to merge all results in a data.frame

```{r}
mses <- result$mse
mses$rmse <- sqrt(mses$mse)
eblups_mse <- merge(eblups, mses, by = "domain")
eblups_mse$CV <- eblups_mse$rmse / eblups_mse$eblup
eblups_mse$RE <- 1.96*eblups_mse$CV
```

We can further merge these results with the stands and plot stand level QMD estimates as maps.

```{r}
eblups_stands <- merge(stands, eblups_mse, by.x = "ID_SMA", by.y = "domain")
estimates_plot <- ggplot() +
  geom_sf(data = eblups_stands, aes(fill = eblup), lwd = 0.5, color = "black") +
  scale_fill_gradient("QMD (cm)    ", low = "white", high = "darkgreen")

RE_plot <- ggplot() +
  geom_sf(data = eblups_stands, aes(fill = RE), lwd = 0.5, color = "black") +
  scale_fill_gradient("Rel error(%)", low = "white", high = "red", labels = scales::label_percent())

grid.arrange(estimates_plot, RE_plot, ncol = 1)
```

Or compare point estimates and uncertainties of direct estimators and eblups. For that we combine in a data frame direct estimates and eblups and create some helper columns.

```{r}
direct_estimates <- group_by(plots_df,ID_SMA)|>
  summarize(QMD_direct=mean(QMD),se_direct = sd(QMD)/sqrt(n()))

eblups_stands <- merge(eblups_stands,direct_estimates,by="ID_SMA")
eblups_stands$unit_lower <- eblups_stands$eblup - 1.96*eblups_stands$rmse
eblups_stands$unit_upper <- eblups_stands$eblup + 1.96*eblups_stands$rmse

eblups_stands$direct_lower <- eblups_stands$QMD_direct - 1.96*eblups_stands$se_direct
eblups_stands$direct_upper <- eblups_stands$QMD_direct + 1.96*eblups_stands$se_direct


scatter_with_whiskers <- ggplot(eblups_stands, aes(x = eblup,y = QMD_direct)) +
  geom_point() + geom_errorbar(aes(ymin=direct_lower,ymax=direct_upper))+
  geom_abline(intercept=0,slope=1)+xlim(10,40)+ylim(10,40)+
  xlab(hat(mu)["U,i"]~(cm))+ylab(hat(mu)["D,i"]~(cm))
scatter_with_whiskers

```

We can compare mean square errors (mses) of direct estimates and eblups as a function of the small area sample size.

```{r}

error_by_n <- pivot_longer(eblups_stands,cols=c("se_direct","rmse"))
error_by_n$Method <- ifelse(error_by_n$name=="rmse","EBLUP","Direct")
ggplot(error_by_n[error_by_n$sampsize>1,], aes(x = sampsize,y = value)) +
  geom_point(aes(shape=Method),color="black") + 
  geom_smooth(aes(lty=Method),color="black")+ theme(legend.position = "bottom")+
  xlab(expression(n[i]))+ylab(expression(rmse~(cm)))


```

# 5. Small area area-level (Fay-Herriot) model:

Area level models are useful in many situations. The most important are:

-   When we don't have plots with good coordinates

-   When the auxiliary information does not allow for precise spatial matching (e.g., GEDI data)

-   When we have measurements that cannot be assimilated to a grid cell. (e.g., transects, variable radius plots, sector plots)

In these situations we can still make use of auxiliary information layers by using area-level models, [**but we have to pay a price in terms of spatial detail**]{.underline}. In area-level models the modeling occurs at the stand level (small area level). We will be able to obtain estimates for stands but not for smaller units.

## Area-level model

The area-level model appears when combining two models. A model relating the true but unknown value of the parameter of interest (e.g., true volume in the stand) and auxiliary information at the stand level, and a model relating ground based estimates for stands and sampling errors.

Let's see a use case of area level models. We are in a ponderosa pine dominated forest in northern California. We have a layer with management units or stands and a set of field plots. In this case the field plots have very good coordinates (we will use them later to do variable selection for a unit-level model), but we are going to assume that the coordinates are not good and we only know the stand in which each plot is located. We also have a set of lidar variables computed over a 20m resolution grid.

```{r}
plots<- read.csv("Plots_BMEF.csv")
stands <- st_read("BMEF.gpkg",layer="stands")
lidar <- rast("BMEF_Metrics.tif")
mapview(lidar[["zq95"]]) + mapview(stands)
plots
```

Some stands were sampled with a variable number of plots and some stands were not sampled. From the plot table we can only know the stand in which each plot was measured.

Before we use the auxiliary information we will estimate the mean volume of the sampled stands knowing that plots inside stands were placed systematically. We will also compute the variance of our estimates.

```{r}
means_vars <- plots |> group_by(StandID) |> 
  summarize(Vol_direct=mean(VOL), Var_Vol_direct = var(VOL)/n(),n=n())
means_vars
```

We cannot work with plot level information so we will summarize auxiliary information to the stand level using zonal statistics. We will compute means and standard deviations for each lidar predictor.

```{r}
mean_aux_stands <- zonal(lidar[[-1]],vect(stands),mean,na.rm=TRUE,as.polygons=TRUE)
names(mean_aux_stands)[-c(1,2)]<-paste("mean",names(mean_aux_stands)[-c(1,2)],sep="_")

sd_aux_stands <- zonal(lidar[[-1]],vect(stands),sd,na.rm=TRUE,as.polygons=TRUE)
names(sd_aux_stands)[-c(1,2)]<-paste("sd",names(sd_aux_stands)[-c(1,2)],sep="_")

aux_info_stands <- merge(mean_aux_stands,sd_aux_stands,by=c("Labelstand","StandID"))
aux_info_stands <- st_as_sf(aux_info_stands)
```

We are going to merge our field estimates with the auxiliary information at the stand level. **Note that stands without ground measurements are removed**.

```{r}
modeling_stands <- merge(means_vars,aux_info_stands,by=c("StandID"))
modeling_stands
```

### Regression model for unknown stand means

We will start assuming that the true average stand volume is related to the mean value of the 95th percentile in the stand.

$$
Vol_i= a + b*meanzq95_i + v_i \ with \ v_i\sim N(0,\sigma^2)
$$

We do not know $Vol_i$ instead we have the means of our plots (direct estimates, $\hat{Vol}_i$).

### Sampling model

Based on sample theory, the mean of the plots within a stand provides an unbiased estimate of the stand volume. For large samples, the sampling error $e_i$ will be normally distributed around 0. With small samples the sample mean remains unbiased but has large variance and an unknown distribution, however, the area-level model is robust to departures from normality.

$$
\hat{Vol}_i = Vol_i + \epsilon_i \ with \ \epsilon_i \sim N(0,\Psi_i^2)
$$

### Area-level model

Combining the two previous equations we obtain

$$
\hat{Vol}_i =a + b*meanzq95_i + v_i + \epsilon_i 
$$

with

$$
v_i \sim N(0,\sigma^2) \  and \ \epsilon_i \sim N(0,\Psi_i^2)
$$

The previous model has two random components bu from sampling theory t we know how to estimate how to estimate the variance $\psi_i^2$ of the sampling error. It is the column Var_Vol_dir in the stand summaries.

### Fitting the area-level model with the `sae` package and stand level estimates

The `sae` package provides two functions for the basic area-level model. They are similar to the ones used for the unit-level model. The first one `eblupFH` computes estimates for the area-level model in sampled stands (FH stands for Fay and Herriot who developed this technique).

```{r}
result_area_level <- eblupFH(Vol_direct ~ mean_zq95, vardir=Var_Vol_direct, 
                             data=modeling_stands)
result_area_level
```

## Uncertainty estimation

The second `mseFH` computes estimates along with uncertainty metrics obtained analytic expressions for the mean squared error. This function also provides stand level estimates. The result is a list with two elements est (another list that contains the summary of the model fit in element fit, and the stand level estimates in eblup) and mse (the mean square error of the stand level estimates).

```{r}
mses_area_level <- mseFH(Vol_direct ~ mean_zq95, vardir=Var_Vol_direct, 
                           data=modeling_stands)
mses_area_level$mse
mses_area_level$est$eblup
```

### Limitations of the `sae` package

The `sae` package only provides estimates and mean square errors for sampled stands. The model can be easily used to obtain synthetic estimates for unsampled stands and formulas for the mean square error in unsampled stands are provided in page XXX of Rao and Molina (2015). We ommit those formulas because they are very complex for this course) in addition, the result from these formulas has to be taken with a grain of salt, at the end we have no information on unsampled stands.

# 6. Variable selection

One problem when working with lidar auxiliary variables is that the number of predictors is typically very large and some variable selection method is needed. In the BMEF example, there are YY predictors. That means that the number of alternative models that we can fit is $2^Y$ which is a huge number. Therefore it is impossible to inspect all the potential models and some algorithm or method is needed. There are many model selection algorithms that we can apply for fixed effects models. A practical approach is to use those methods to obtain a reduced list of candidate models and then inspect those models gradually.

Below you have a template of a model selection method that uses three steps:

1.  Elimination of highly correlated variables and variables with low variability.

2.  Best subset selection to generate a list of candidate models with up to 6 auxiliary variables. For each number of auxiliary variables included in the model (1,2,...,6) the best 5 models are selected. That results in a list of 30 models that we can sort by adjusted r-square.

3.  We move through the list of candidate models fitting the unit-level or area-level model to our data and inspect the coefficients and residuals. If something is not appropriate we discard the model and move to the next one on the list of candidates.

We recommend doing the last step manually or at least using some graphical summaries to assist the process. Step 3 is the step in which we will check our model, remember that these techniques are based on the model, i.e., the model must hold, so it is importnat to be patient and thorough in this step.

[**Step 1. Removal of highly correlated variables and variables with no variation**]{.underline}

```{r}
data <- st_read("BMEF.gpkg",layer="plots")
data <- st_drop_geometry(data) 
predictors <- c(10:57)
X<-data[,predictors]

nzv_predictors <- caret::nearZeroVar(X)
X <-X[,-nzv_predictors]
corr_matrix_predictors <- cor(X)
high_corr_indices <- caret::findCorrelation(corr_matrix_predictors, 
                                            cutoff = 0.95, exact = FALSE)
reduced_predictors_matrix <- corr_matrix_predictors[, -high_corr_indices]
X <- X[, -high_corr_indices]

Y <- data[,"VOL",drop=FALSE]

```

[**Step 2. Run best subsets (add vif and initialize to NA and sort by r-squared)**]{.underline}

```{r}
select <- regsubsets(X,Y[,1],nbest = 5,nvmax = 6,really.big = TRUE)
summary_select <- summary(select)
models <- apply(summary_select$which, 1, function(y) names(y)[which(y)][-1])
summary_select$vif<-summary_select$adjr2
summary_select$vif[]<-NA
summary_select$order<-1:length(summary_select$rsq)
n <- rowSums(summary_select$which)
```

[**Step 3-a. Tag and remove models with large vif and test if error variance increases.**]{.underline}

```{r}
models_list <- list()

for(i in 1:dim(summary_select$which)[1]){
  # print(i)
  formula_i <- as.formula(paste0(colnames(Y),"~",paste(models[[i]],collapse="+")))
  # print(models[[i]])
  X_i<-X[ ,models[[i]],drop=FALSE]
  data_i <- cbind(Y,X_i)
  # print(names(data_i))
  model<-try(gls(formula_i,data= data_i,weights=varExp(0.1,form=~fitted(.)),
                control=glsControl(maxIter = 5000,msMaxIter = 5000,opt="optim")))
  lm_model<-lm(formula_i,data= data_i)
  if(inherits(model,"try-error")){
    model<-lm_model
  }
  if(inherits(model,"gls")){
    a<-summary(model)$tTable
  }else{
    a<-summary(model)$coefficients
  }
  if(dim(X_i)[2]==1){
    # print(0)
    if(all(a[,4]<0.05)){
      summary_select$vif[i] <- 0
    }else{
      summary_select$vif[i] <- 15
    }
  }else{
    # print(x)
    if(all(a[,4]<0.05)){
      summary_select$vif[i] <- max(vif(model))
    }else{
      summary_select$vif[i] <- 15
    }
  }
  models_list[[i]]<-list(model=model,lm=lm_model)
}

adjr2 <-summary_select$adjr2[summary_select$vif<5]
order_model <- summary_select$order[summary_select$vif<5]

```

[**Step 3-b. Sort by adjusted r-squared and start testing manually.**]{.underline}

```{r}
selected <- order_model[which.max(adjr2)]
selected <- models_list[[selected]]
if(inherits(selected$model,"gls")){
  anova_models <- anova(selected$model,selected$lm)
  if(anova_models$`p-value`[2]<0.05){
    test_model<-selected$model
  }else{
     test_model<-selected$lm
  }
    
}else{
  test_model<-selected$lm
}
  
test_model
summary(test_model)
```

# 7. References

Pinheiro, J.C., Bates, D.M., 2000. Mixed-Effects Models in S and S-PLUS. Springer, New York. <https://doi.org/10.1007/b98882>

Rao, J.N.K., Molina, I., 2015. Small Area Estimation. John Wiley & Sons, Inc.

Särndal, C.-E., Swensson, B., Wretman, J., 2003. Model Assisted Survey Sampling (Springer Series in Statistics).