pairwise PERMDISP no F values

[bark9299]

I am using the commands:

permdisp_logCPM_wu_body <- betadisper(wu_dist_logCPM_body_site, ps_lcpm_40k_df$body.site)
test_permdisp <- permutest(permdisp_logCPM_wu_body, pairwise = TRUE)

But the resulting table only give p-values for each pairwise comparison, but not F. Is there a way to see what the F value is for all these pairwise comparisions?

Thanks

[jarioksa]

Strange thing is that when I use the same commands, I get both a table of p-values and a table of F statistics. Can you prepare a reproducible example, where 'reproducible' means that we can reproduce the example. Using, say, data in vegan is a good choice so that you do not need to provide your own data (and I usually do not want to download alien data).

> m <- with(dune.env, betadisper(vegdist(dune), Management))
> permutest(m, pairwise = TRUE)

Permutation test for homogeneity of multivariate dispersions
Permutation: free
Number of permutations: 999

Response: Distances
          Df  Sum Sq  Mean Sq      F N.Perm Pr(>F)
Groups     3 0.13831 0.046104 1.9506    999   0.15
Residuals 16 0.37816 0.023635                     

Pairwise comparisons:
(Observed p-value below diagonal, permuted p-value above diagonal)
         BF       HF       NM    SF
BF          0.570000 0.058000 0.267
HF 0.593549          0.076000 0.351
NM 0.060191 0.064936          0.372
SF 0.281235 0.402333 0.365221      

[bark9299]

Hi @jarioksa,

What I meant was that I don't get a F statistic for every pairwise comparision. Using the dune data example that you showed, I would like to know the F statistic of HF x BF, HF x NM, HF x SF.... It only gives you the overall F statisitic for all groups. Thanks

[jarioksa]

Why? Pairwise tests use t-statistics. As a first trick, you may try F _1,p = t_p². Nothing of this makes sense in permutation tests that do not use theoretical distributions, but are based on ... permutations. I suggest you consult your statistical text book, and prepare to explain to referees why you used F statistics in pairwise tests instead of t statistics.

[bark9299]

You are right I am not extremely versed on statistics but I was following the way Qiime2 outputs pairwise permdisp. Qiime reports the F statistic for every pairwise comparision. Are you saying Qiime2 is wrong?

[gavinsimpson]

As a F statistic for a single term (e.g. an F test for a linear model with a single covariate - try it with a lm() and look at the F and t stat in the output) is directly comparable to a t statistic as @jarioksa showed ($F_{1,p} = t_{p}^2$; i.e. an F stat with 1 and $p$ degrees of freedom is equal to the squared t statistic with $p$ degrees of freedom), we're obviously not saying Qiime2 is wrong or right. Just stating a statistical fact.

[gavinsimpson]

If you want them, the observed statistics (F for the omnibus test, t for each pairwise test) are available in the statistic component of the object returned by permutest.betadisper():

# load data
data(varespec)

# Hellinger distances between samples
dis <- vegdist(varespec, method = "hellinger")

# First 16 sites grazed, remaining 8 sites ungrazed
groups <- factor(c(rep(1,16), rep(2,8)), labels = c("grazed","ungrazed"))

# Calculate multivariate dispersions
mod <- betadisper(dis, groups)
mod

# Perform test
anova(mod)

# Permutation test for F
pmod <- permutest(mod, permutations = 99, pairwise = TRUE)

# extract the observed test statistics
pmod$statistic

which gives

r$> pmod$statistic
        Overall (F) grazed-ungrazed (t) 
           5.759668            2.399931

[gavinsimpson]

Or perhaps more interestingly (using @jarioksa's example):

dij <- vegdist(dune, method = "hellinger")
m <- with(dune.env, betadisper(dij, Management))
ptest <- permutest(m, pairwise = TRUE)
ptest$statistic

yielding

r$> ptest$statistic
Overall (F)   BF-HF (t)   BF-NM (t)   BF-SF (t)   HF-NM (t)   HF-SF (t)   NM-SF (t) 
  1.5728727  -0.2813427  -2.2134247  -1.0345128  -1.8319082  -0.7853948   1.0470113

[gavinsimpson]

And of course, if you must, you can take the square root of the absolute value of the $t$ statistic to get $F$ statistics, but I really don't see the point, and you loose the sign this way:

> sqrt(abs(ptest$statistic[-1]))
BF-HF (t) BF-NM (t) BF-SF (t) HF-NM (t) HF-SF (t) NM-SF (t) 
0.5304175 1.4877583 1.0171100 1.3534800 0.8862251 1.0232357 

The sign is useful because it tells you which group in each pair has the larger dispersion.

[jarioksa]

Like written in scriptures, F-values are squares of t, and you can always calculate F from t. In a way it is only a convention which one to use. I think the main reasons for using t instead of F is that t has sign and t is two-sided: you see if the comparison means negative or positive difference. The code in permutest.betadisper uses 2 * pt(-abs(T0), df) to get P from the theoretical distribution (as there are two tails). F is always positive and you only have information of difference, but not of the direction of the difference. Would I develop Qiime I would use t, but it is not wrong to use F, in particular as Qiime seems to base their P values on permutations, or at least they are not the analytic values from the theoretical F distribution, and there t and F give same results. Consult R help(pf) for the relationship between F and t.