Analysing the False Discovery Proportion in Competition-Based Multiple Hypothesis Testing

Abstract

The commonly practised modern approach to analysing a large number of hypotheses is to control the false discovery rate (FDR), the expected value of the false discovery proportion (FDP). Several such popular procedures are available when p-values can be assigned to each hypothesis, but the mass spectrometry community had encountered a problem where such p-values were not readily available. To resolve this issue, they developed target-decoy competition (TDC), a competition-based approach to controlling the FDR where, instead of a p-value, each observation (referred to as a target score) is paired with a competing null-generated score (referred to as a decoy score). The paired target and decoy scores are each competed and only the winning score, as well as a target/decoy-win label, is kept. Such scores and labels allow us to estimate, and hence control, the FDR.

Controlling the FDR among a set of rejected hypotheses, however, only controls the FDP in that set in an average sense. While we can enforce the FDR to be appropriately bounded, the actual FDP may undesirably exceed our desired tolerance. Arguably, it makes more sense to gauge the FDP in our list of discoveries instead of an expected value that could only be obtained through an infinite number of experiment repetitions. Hence, we interest ourselves in the analysis of the FDP for competition-based multiple hypothesis testing.

We examine in this thesis three major topics. First, we introduce ways to give an upper confidence bound on the FDP in the list of discoveries generated using the FDR as the chosen error metric. Second, we introduce ways to derive a list of discoveries in which the FDP is directly controlled with some pre-specified confidence. Third, and finally, we introduce ways to derive a similar list in cases where the set of hypotheses can be partitioned into groups possessing distinct characteristics.