Skip to content

Latest commit

 

History

History
74 lines (66 loc) · 4.44 KB

1_methods.md

File metadata and controls

74 lines (66 loc) · 4.44 KB

Methods

Core Methods

  • 🔥 IHW (Independent Hypothesis Weighting) 🔥
    • 💡Idea: Method uses a data-driven approach to calculate weights using independent covariates and then applies a group-weighted BH method
    • Features Size investing
    • Assumptions:
      • covariate must be independent of p-value under null
      • tests are independent under null
    • 👍Good: Improves power and controls FDR at alpha level
    • 👎Bad: does not maintain FDR when (1) insufficient power to detect false hypotheses (2) all hypotheses are true
  • 🔥 ASH (Adaptive SHrinkage) 🔥
    • 💡Idea:
    • 👍Good:
    • 👎Bad:
  • 🔥 Boca-Leek 🔥
    • 💡Idea:
      • Estimates FDR conditional on covariates in a multiple testing framework by (1) estimating the null proportion of hypotheses with logistic regression and (2) multiplying the BH-adjusted p-values by the estimated null proportion.
      • Assumes that the p-values are independent of the covariates conditional on the null or alternative. In other words, the probability that a test statistic or p-value is drawn from the null or non-null distribution depends on the covariates (but not the realized value of the test statistic or p-value itself).
    • 👍Good:
      • FDR control can be maintained even when tests are moderately correlated
      • If hypotheses are independent, can use bootstrapping to obtain CIs for the null proportion of hypotheses
      • Increasing the number of tests can lead to improvement in FDR control
    • 👎Bad:
      • Improvement over Storey (2002) is minor
      • Scott (2015) is superior when test statistics are normally distributed
      • Does not control FDR when hypotheses are highly correlated
      • Requires tuning of $\lambda$ parameter and careful specification of covariates (smoothing splines? categorization?)

Relevant Methods

  • Benjamini and Hochberg (BH)
    • 💡Idea: Uses p-values. Assumes independence of tests (or assumes all hypotheses are exchangeable), but has been shown to be robust even with correlated tests.
    • 👍Good: Easy to use. More powerful than FWER-based methods
    • 👎Bad: Sub-optimal power (and can't prioritize tests) when the individual tests differ in statistical properties such as sample size, true effect size, signal-to-noise ratio or prior probability of being false (see Ignatiadis et al. 2016)
      • To increase power while controlling FDR, can use independent covariates to prioritize tests
  • Weighted BH (basis of IHW)
    • 💡Idea: Associate each test with a non-negative weight such that weights averahe to 1. Hypotheses with higher weights are prioritized.
    • 👍Good: Controls FDR
    • 👎Bad: Weights must be prespecified, are independent of data
  • Greedy Independent Filtering (compared in IHW paper)
    • 💡Idea: Filter all p-values using an independent covariate ($X$) such that $X$ is less than some threshold $x$. Greedy if researcher tests all possible thresholds and picks one that maximizes number of discoveries. However, greedy approaches do not control for FDR at alpha level.
    • 👍Good: Controls FDR at alpha level IF researcher ONLY uses a pre-specified threshold
    • 👎Bad: Not automated, very subjective, can lead to p-hacking
  • LSL-GBH, TST-GBH (compared in IHW paper)
    • 💡Idea:
    • 👍Good:
    • 👎Bad:
  • SBH (compared in IHW paper)
    • 💡Idea: Use a categorial (or binned continuous) covariate to stratify tests, apply BH within each strata, combine significant tests
    • 👍Good:
    • 👎Bad: Loss of FDR control for the null tests because different strata are treated equally
      • To increase power, different strata should be prioritized differently
  • Clfdr (Cai's local FDR) (compared in IHW paper)
    • 💡Idea:
    • 👍Good:
    • 👎Bad:
  • FDRreg (compared in IHW paper, swfdr paper)
    • 💡Idea:
    • 👍Good:
    • 👎Bad:
  • local false discovery rate (compared in ASH paper)
    • 💡Idea: local FDR is estimated separately within each group and then estimates are grouped together
    • 👍Good:
    • 👎Bad: FDR isn't controlled at alpha level if alternative distributions across groups are different
  • mixture false discovery rate (compared in ASH paper)
    • 💡Idea:
    • 👍Good:
    • 👎Bad: