E-values: Theory and methods

Click this link for a very small review article (updated June 2023) on e-values and e-processes.
Click this link for an e-book (draft) co-authored with Aaditya Ramdas.

The series of working papers contains recent advances on the theory and methods of e-values and e-processes. Here, e in e-values stands for expectation whereas p in p-values stands for probability. E-values have many advantages over p-values. In particular, they are convenient for optional stopping in data collection and experiment design, arbitrary dependence in multiple testing, and high-dimensional composite hypotheses. E-values are intimately linked to martingales, likelihood ratios, and betting scores (Glenn Shafer); see details in WP01. This working paper series has an overlap with Hypothesis testing with e-values maintained by Vladimir Vovk. See also the work of Peter Grünwald and Aaditya Ramdas. Some basic terminology and their references are listed below.

A brief description of each working paper is provided below to explain its main results and the logical structure across papers.

We offer improved thresholds for e-values over Markov's inequality under distributional information. For many common assumptions, the required threshold can be divided by a factor of 2 or exp(1). The main results can be seen as generalizations of Markov's inequality.

With the help of e-values we obtain merging methods for vectors of exchangeable p-values. They strictly improve the classic methods in a few ways. For example, we derive randomized and/or exchangeable improvements of well known rules like "twice the median" and "twice the average", as well as geometric and harmonic means.

We construct generic e-processes for testing conditional mean and conditional variance, with optional information on the distributional shape of the test statistics. The journal version is to appear in Biometrika (2024).

We study the existence and construction of exact p-values, e-values and test martingales for composite null and composite alternative. The main result is Theorem 7.3 which establishes necessary and sufficient conditions. The proofs rely on recently developed techniques in simultaneous optimal transport. We also propose the SHINE construction method for exact p-values and e-values. The journal version is to appear in Annals of Statistics (2024).

The multiple testing literature has primarily dealt with three types of dependence assumptions between p-values: independence, positive regression dependence, and arbitrary dependence. We provide new results about negatively dependent p-values and e-values, and provide several examples as to when negative dependence may arise. The journal version is to appear in Bernoulli (2024).

One of the most challenging tasks in risk modeling practice is to backtest ES (see the working paper series on Axiomatic theory of risk measures for a background on risk measures) forecasts provided by financial institutions. We use e-values and e-processes to construct a model-free backtesting procedure for ES using a concept of universal e-statistics, which can be naturally generalized to many other risk measures and statistical quantities.

We consider testing the nonparametric hypothesis of symmetry, introduce analogues for e-values of three popular nonparametric tests and define an analogue for e-values of Pitman’s asymptotic relative efficiency. The notion of e-power is formally studied. The journal version is published in New England Journal of Statistics in Data Science (2024).

We study procedures based on both e-values and p-values for each hypothesis to show the advantages of using e-values as random weights. We also collect several other results, such as a tiny but uniform improvement of e-BH, a soft-rank permutation e-value, and the use of e-values as masks in interactive multiple testing. The journal version is to appear in Biometrika (2023).

This is a follow-up of WP06. Inspired by the Benjamini-Yekutieli (BY) procedure for reporting confidence intervals, we design the e-BY procedure which uses e-CIs instead of p-CIs. The e-BY procedure has a desirable false coverage rate control under arbitrary dependence and stopping rules, in contrast to the BY procedure. The journal version is published in Electronic Journal of Statistics (2024).

This is a follow-up of WP06. We apply e-values and the e-BH procedure to multi-armed bandit testing problems. We propose a unified, modular framework for FDR control that emphasizes the decoupling of exploration and summarization of evidence. We utilize the powerful e-processes to ensure FDR control for arbitrary composite nulls, exploration rules and stopping times in generic problem settings. The conference version is published in Advances in Neural Information Processing Systems (NeurIPS) (2021).

We introduce the notion of p*-values (p*-variables), which generalizes p-values (p-variables) in several senses. The new notion has four natural interpretations: operational, probabilistic, Bayesian, and frequentist. A main example of a p*-value is a mid p-value. A unified stochastic representation (Theorem 3.1) for p-values, mid p-values, and p*-values is obtained. The notion of p*-values becomes useful in many situations even if one is only interested in p-values, mid p-values, or e-value; it is connected to e-values in a simple way: 1/(2e) is a p*-value for an e-value e. The journal version is published in Bernoulli (2024).

We design a natural analog of the Benjamini-Hochberg (BH) procedure for false discovery control (FDR) control that utilizes e-values (e-BH) and compare it with the standard procedure for p-values. The e-BH procedure includes the BH procedure as a special case through calibration between p-values and e-values. The journal version is published in Journal of the Royal Statistical Society Series B (2022).

This paper is a follow-up of WP12 of Joint mixability and negative dependence. We study the admissibility in Wald's sense of p-merging functions and their domination structure under arbitrary dependence. As a technical tool e-values become essential in the main results (Theorems 5.1 and 5.2). Another result (Theorem 6.2) showing the admissibility of some p-merging methods uses results on Joint mixability and negative dependence. The journal version is published in Annals of Statistics (2022).
- Erratum in Eq. (21) of the journal version: f(1)=0 should be f(τ)=0.

This is a follow-up of WP01. We study the problem of merging sequential or independent e-values into one e-value for statistical decision making. We describe a class of e-value merging functions via martingale merging functions, and show that all merging methods for sequential e-values are dominated by such a class. In case of merging independent e-values, the situation becomes much more sophisticated, and we provide a general class of such merging functions based on reordered test martingales. The journal version is published in Electronic Journal of Statistics (2024).

This is a follow-up of WP02. We use e-discovery matrices in the context of multiple hypothesis testing assuming that the base tests produce independent e-values, and compare those with procedures based on p-values. The journal version is to appear in Canadian Journal of Statistics (2024).

We discuss systematically two versions of confidence regions: those based on p-values and those based on e-values, the latter are e-confidence intervals (e-CIs). Both versions are applied to multiple hypothesis testing to construct procedures that control the number of false discoveries under arbitrary dependence between the base p- or e-values. We introduce the e-discovery matrices and show that it is efficient both computationally and statistically using simulated and real-world datasets. The journal version is published in Statistical Science (2023).

We introduce e-values in their pure form to the statistical community. In particular, we demonstrate that e-values are often mathematically more tractable and develop procedures using e-values for multiple testing of a single hypothesis and testing multiple hypotheses. Related concepts introduced in this paper are e-variables, sequential e-values, and e-merging functions. The journal version is published in Annals of Statistics (2021).