7062:
journal publications referenced below sometimes coming years later). In these, the concept was finally given a proper name ("S-Value" and "E-Value"; in later versions of their paper, also adapted "E-Value"); describing their general properties, two generic ways to construct them, and their intimate relation to betting). Since then, interest by researchers around the world has been surging. In 2023 the first overview paper on "safe, anytime-valid methods", in which e-values play a central role, appeared.
6522:", and so on. With e-processes, we obtain an e-variable with any such rule. Crucially, the data analyst may not know the rule used for stopping. For example, her boss may tell her to stop data collecting and she may not know exactly why - nevertheless, she gets a valid e-variable and Type-I error control. This is in sharp contrast to data analysis based on p-values (which becomes invalid if stopping rules are not determined in advance) or in classical Wald-style
794:. But, whereas with standard p-values the inequality (*) above is usually an equality (with continuous-valued data) or near-equality (with discrete data), this is not the case with e-variables. This makes e-value-based tests more conservative (less power) than those based on standard p-values, and it is the price to pay for safety (i.e., retaining Type-I error guarantees) under optional continuation and averaging.
1044:. Thus, when the null is simple, e-variables coincide with likelihood ratios. E-variables exist for general composite nulls as well though, and they may then be thought of as generalizations of likelihood ratios. The two main ways of constructing e-variables, UI and RIPr (see below) both lead to expressions that are variations of likelihood ratios as well.
54:, are the fundamental building blocks for anytime-valid statistical methods (e.g. confidence sequences). Another advantage over p-values is that any weighted average of e-values remains an e-value, even if the individual e-values are arbitrarily dependent. This is one of the reasons why e-values have also turned out to be useful tools in
7061:
in various papers with various collaborators (e.g.), and an independent re-invention of the concept in an entirely different field, the concept did not catch on at all until 2019, when, within just a few months, several pioneering papers by several research groups appeared on arXiv (the corresponding
5527:
Finally, in practice, one sometimes resorts to mathematically or computationally convenient combinations of RIPr, UI and other methods. For example, RIPr is applied to get optimal e-variables for small blocks of outcomes and these are then multiplied to obtain e-variables for larger samples - these
5434:
The advantage of the UI method compared to RIPr is that (a) it can be applied whenever the MLE can be efficiently computed - in many such cases, it is not known whether/how the reverse information projection can be calculated; and (b) that it 'automatically' gives not just an e-variable but a full
45:
In contrast to p-values, e-values can deal with optional continuation: e-values of subsequent experiments (e.g. clinical trials concerning the same treatment) may simply be multiplied to provide a new, "product" e-value that represents the evidence in the joint experiment. This works even if, as
5478:
Its main disadvantage compared to RIPr is that it can be substantially sub-optimal in terms of the e-power/GRO criterion, which means that it leads to tests which also have less classical statistical power than RIPr-based methods. Thus, for settings in which the RIPr-method is computationally
3105:
e-value: it is an e-variable by definition, but it will never allow us to reject the null hypothesis. This example shows that some e-variables may be better than others, in a sense to be defined below. Intuitively, a good e-variable is one that tends to be large (much larger than 1) if the
46:
often happens in practice, the decision to perform later experiments may depend in vague, unknown ways on the data observed in earlier experiments, and it is not known beforehand how many trials will be conducted: the product e-value remains a meaningful quantity, leading to tests with
5666:, and this domination is strict if the inequality is strict. An admissible calibrator is one that is not strictly dominated by any other calibrator. One can show that for a function to be a calibrator, it must have an integral of at most 1 over the uniform probability measure.
4324:". The method of mixtures essentially amounts to "being Bayesian about the numerator" (the reason it is not called "Bayesian method" is that, when both null and alternative are composite, the numerator may often not be a Bayes marginal): we posit any prior distribution
5423:
5096:
1373:. If the null is composite, then some special e-variables can be written as Bayes factors with some very special priors, but most Bayes factors one encounters in practice are not e-variables and many e-variables one encounters in practice are not Bayes factors.
2458:
as large as possible in the "e-power" or "GRO" sense (see below). Waudby-Smith and Ramdas use this approach to construct "nonparametric" confidence intervals for the mean that tend to be significantly narrower than those based on more classical methods such as
4058:
5487:. However, in many other statistical testing problems, it is currently (2023) unknown whether fast implementations of the reverse information projection exist, and they may very well not exist (e.g. generalized linear models without the model-X assumption).
650:
1505:. Based on this interpretation, the product e-value for a sequence of tests can be interpreted as the amount of money you have gained by sequentially betting with pay-offs given by the individual e-variables and always re-investing all your gains.
61:
E-values can be interpreted in a number of different ways: first, the reciprocal of any e-value is itself a p-value, but a special, conservative one, quite different from p-values used in practice. Second, they are broad generalizations of
5919:
7294:
Bacillus
Calmette-Guérin vaccine to reduce COVID-19 infections and hospitalisations in healthcare workers – a living systematic review and prospective ALL-IN meta-analysis of individual participant data from randomised controlled
3901:
5523:
type. However, while these superficially look very different from likelihood ratios, they can often still be interpreted as such and sometimes can even be re-interpreted as implementing a version of the RIPr-construction.
4470:
5479:
feasible and leads to e-processes, it is to be preferred. These include the z-test, t-test and corresponding linear regressions, k-sample tests with
Bernoulli, Gaussian and Poisson distributions and the logrank test (
2603:
of each e-value is allowed to depend on all previous outcomes, and no matter what rule is used to decide when to stop gathering new samples (e.g. to perform new trials). It follows that, for any significance level
2471:
E-values are more suitable than p-value when one expects follow-up tests involving the same null hypothesis with different data or experimental set-ups. This includes, for example, combining individual results in a
1055:. Importantly, neither (a) nor (b) are e-variables in general: generalized likelihood ratios in sense (a) are not e-variables unless the alternative is simple (see below under "universal inference"). Bayes factors
3572:
have densities (denoted by lower-case letters) relative to the same underlying measure. Grünwald et al. show that under weak regularity conditions, the GRO e-variable exists, is essentially unique, and is given by
2271:
6550:-aggressive rule is always allowed. Because of this validity under optional stopping, e-processes are the fundamental building block of confidence sequences, also known as anytime-valid confidence intervals.
1112:
74:. Interest in e-values has exploded since 2019, when the term 'e-value' was coined and a number of breakthrough results were achieved by several research groups. The first overview article appeared in 2023.
6526:(which works with data of varying length but again, with stopping times that need to be determined in advance). In more complex cases, the stopping time has to be defined relative to some slightly reduced
4235:
3530:
3110:
an alternative is available, we would like them to be small (p-values) or large (e-values) with high probability. In standard hypothesis tests, the quality of a valid test is formalized by the notion of
3638:
6849:
4277:
have densities relative to the same underlying measure. There are now two generic, closely related ways of obtaining e-variables that are close to growth-optimal (appropriately redefined for composite
1246:
4870:
2037:
6958:
5776:
6732:
6366:
5718:
5280:
4945:
2423:
5237:
5158:
4913:
4683:
3916:
7292:
Ter Schure, J.A. (Judith); Ly, Alexander; Belin, Lisa; Benn, Christine S.; Bonten, Marc J.M.; Cirillo, Jeffrey D.; Damen, Johanna A.A.; Fronteira, Inês; Hendriks, Kelly D. (2022-12-19).
4781:
2324:
950:
188:
6391:, based only on the data observed so far, whether to stop collecting data or not. For example, this could be "stop when you have seen four consecutive outcomes larger than 1", "stop at
1647:
414:
4595:
2799:
2597:
2539:
1797:
5989:
While of theoretical importance, calibration is not much used in the practical design of e-variables since the resulting e-variables are often far from growth-optimal for any given
4716:
4534:
4099:
6208:
3757:
3670:
1371:
7049:
Historically, e-values implicitly appear as building blocks of nonnegative supermartingales in the pioneering work on anytime-valid confidence methods by well-known mathematician
553:
7039:
6780:
6520:
6254:
6123:
6073:
4641:
3266:
513:
5983:
2634:
7053:
and some of his students. The first time e-values (or something very much like them) are treated as a quantity of independent interest is by another well-known mathematician,
6294:
4275:
3570:
3418:
3320:
1307:
1018:
846:
768:
6902:
5594:
5273:
5194:
2456:
2159:
1541:
2106:
6993:
5521:
4369:
4170:
3808:
3465:
3359:
3200:
1735:
1702:
2346:
1617:
546:
366:
3065:
2953:
2714:
2666:
6869:
6851:
is a test supermartingale, and hence also an e-process (note that we already used this construction in the example described under "e-values as bets" above: for fixed
2126:
6548:
6459:
6437:
6156:
5796:
5664:
4807:
3026:
2897:
2734:
2686:
1673:
1152:
792:
705:
5804:
1584:
1503:
1414:
333:
6617:
6590:
6415:
6014:
5473:
4940:
4302:
3727:
3145:
2864:
2837:
1910:
1883:
1824:
1334:
734:
235:
215:
110:
3099:
1950:
1480:
680:
6479:
6389:
5638:
5618:
5453:
4342:
4119:
3781:
3696:
3220:
2993:
2973:
2921:
2179:
1970:
1930:
1561:
1454:
1436:
is an e-variable" and "if the null hypothesis is true, you do not expect to gain any money if you engage in this bet" are logically equivalent. This is because
1434:
1172:
1132:
1042:
972:
886:
866:
481:
437:
283:
255:
130:
1856:
2975:
can be decomposed as a product of per-outcome e-values in this way though. If this is not possible, we cannot use them for optional stopping (within a sample
5924:
Conversely, an e-to-p calibrator transforms e-values back into p-variables. Interestingly, the following calibrator dominates all other e-to-p calibrators:
3813:
3106:
alternative is true. This is analogous to the situation with p-values: both e-values and p-values can be defined without referring to an alternative, but
2743:
Mathematically, this is shown by first showing that the product e-variables form a nonnegative discrete-time martingale in the filtration generated by
5494:
settings (such as testing a mean as in the example above, or nonparametric 2-sample testing), it is often more natural to consider e-variables of the
5475:, the resulting ratio is still an e-variable; for the reverse information projection this automatic e-process generation only holds in special cases.
4376:
450:(a number) is often used when one is really referring to the underlying e-variable (a random variable, that is, a measurable function of the data).
5275:
as an alternative). Note in particular that when using the plug-in method together with the UI method, the resulting e-variable will look like
2186:
70:. Third, they have an interpretation as bets. Finally, in a sequential context, they can also be interpreted as increments of nonnegative
1062:
6557:, which are nonnegative supermartingales with starting value 1: any test supermartingale constitutes an e-process but not vice versa.
38:(e.g., "the coin is fair", or, in a medical context, "this new treatment has no effect"). They serve as a more robust alternative to
7330:
6736:(again, in complex testing problems this definition needs to be modified a bit using reduced filtrations). Then the product process
3268:; in case of composite alternatives, there are various versions (e.g. worst-case absolute, worst-case relative) of e-power and GRO.
2480:
Indeed, they have been employed in what may be the world's first fully 'online' meta-analysis with explicit Type-I error control.
4175:
3470:
7852:
3578:
6785:
1179:
4820:
7483:
1975:
6907:
5723:
5418:{\displaystyle {\frac {\prod _{i=1}^{n}q_{{\breve {\theta }}\mid X^{i-1}}(X_{i})}{q_{{\hat {\theta }}\mid X^{n}}(X^{n})}}}
5091:{\displaystyle {\bar {q}}_{\breve {\theta }}(X^{n})=\prod _{i=1}^{n}q_{{\breve {\theta }}\mid X^{i-1}}(X_{i}\mid X^{i-1})}
6625:
6299:
2812:
We already implicitly used product e-variables in the example above, where we defined e-variables on individual outcomes
2802:
5672:
6963:
Another way to construct an e-process is to use the universal inference construction described above for sample sizes
4053:{\displaystyle E={\frac {q(Y)}{\sup _{P\in H_{0}}p(Y)}}\left(={\frac {q(Y)}{{p}_{{\hat {\theta }}\mid Y}(Y)}}\right)}
2688:. Thus if we decide to combine the samples observed so far and reject the null if the product e-value is larger than
2351:
6076:
5199:
5120:
4875:
1047:
Two other standard generalizations of the likelihood ratio are (a) the generalized likelihood ratio as used in the
4646:
6527:
4721:
2801:(the individual e-variables are then increments of this martingale). The results then follow as a consequence of
2279:
893:
135:
71:
1622:
83:
27:
373:
7857:
4810:
4542:
4063:
2746:
2544:
2486:
1744:
645:{\displaystyle P\left(E\geq {\frac {1}{\alpha }}\right)=P(1/E\leq \alpha )\ {\overset {(*)}{\leq }}\ \alpha }
5427:
which resembles, but is still fundamentally different from, the generalized likelihood ratio as used in the
4688:
4477:
4069:
6161:
3732:
3645:
1456:
being an e-variable means that the expected gain of buying the ticket is the pay-off minus the cost, i.e.
1339:
55:
7862:
6998:
6739:
6484:
6213:
6082:
6032:
4600:
3225:
486:
7057:, in 1976, within the theory of algorithmic randomness. With the exception of contributions by pioneer
5930:
4321:
2607:
2927:(not just "batch of outcomes") we like, for whatever reason, and reject if the product so far exceeds
6266:
4240:
3535:
3364:
3279:
1253:
977:
805:
771:
739:
47:
6874:
5543:
5242:
5163:
2428:
2131:
1511:
3729:. Under further regularity conditions (and in all practically relevant cases encountered so far),
2161:
is guaranteed to be nonnegative). We may then define a new e-variable for the complete data vector
1738:
685:
6371:
In basic cases, the stopping time can be defined by any rule that determines, at each sample size
2042:
6966:
5497:
4347:
4136:
3786:
3431:
3325:
3166:
1711:
1678:
3151:
in economics and (since it does exhibit close relations to classical power) is sometimes called
2329:
1589:
518:
338:
5914:{\displaystyle \int _{0}^{1}\kappa p^{\kappa -1}d\kappa ={\frac {1-p+p\log p}{p(-\log p)^{2}}}}
5480:
3673:
3031:
2930:
2691:
2643:
6854:
2839:
and designed a new e-value by taking products. Thus, in the example, the individual outcomes
2111:
6533:
6444:
6422:
6128:
5781:
5643:
5428:
4786:
2998:
2869:
2806:
2719:
2671:
1652:
1137:
1048:
777:
690:
7293:
3423:
1566:
1485:
1384:
303:
7080:
6595:
6568:
6394:
5992:
5458:
5196:) with the main methods for the composite null (UI or RIPr, using the single distribution
5117:
settings, we can simply combine the main methods for the composite alternative (obtaining
4918:
4280:
3705:
3123:
2842:
2815:
1888:
1861:
1802:
1312:
712:
220:
193:
88:
7540:"Panning for Gold: 'Model-X' Knockoffs for High Dimensional Controlled Variable Selection"
3078:
1935:
8:
6523:
5483:
is available for a subset of these), as well as conditional independence testing under a
1459:
657:
63:
7643:
1708:
nulls, writing it as a likelihood ratio is usually mathematically more convenient. The
1508:
The betting interpretation becomes particularly visible if we rewrite an e-variable as
7810:
7699:
7659:
7603:
7551:
7399:
7256:
7162:
7092:
6464:
6374:
5623:
5603:
5438:
4327:
4104:
3766:
3681:
3205:
2978:
2958:
2906:
2164:
1955:
1915:
1546:
1439:
1419:
1157:
1117:
1027:
957:
871:
851:
466:
422:
268:
240:
115:
7774:
7739:
6565:
E-processes can be constructed in a number of ways. Often, one starts with an e-value
4125:
method, "universal" referring to the fact that no regularity conditions are required.
2483:
Informally, optional continuation implies that the product of any number of e-values,
1829:
7828:
7779:
7761:
7717:
7663:
7621:
7569:
7520:
7479:
7427:
7419:
7365:
7274:
7222:
7180:
7110:
5536:
There exist functions that convert p-values into e-values. Such functions are called
3896:{\displaystyle p_{\curvearrowleft Q}(Y)=\int _{\Theta _{0}}p_{\theta }(Y)dW(\theta )}
7590:
Shafer, Glenn; Shen, Alexander; Vereshchagin, Nikolai; Vovk, Vladimir (2011-02-01).
7820:
7769:
7751:
7709:
7655:
7613:
7561:
7512:
7471:
7409:
7357:
7298:
7266:
7214:
7172:
7102:
4814:
3760:
1972:
are made. Then we may first construct a family of e-variables for single outcomes,
1705:
5101:
Effectively, both the method of mixtures and the plug-in method can be thought of
4121:
is always well-defined). This way of constructing e-variables has been called the
257:, which represent the full sequence of outcomes of a statistical experiment, as a
7446:
7050:
6554:
3148:
3113:
35:
7798:
7463:
7302:
3911:
In the same setting as above, show that, under no regularity conditions at all,
3120:
The standard notion of quality of an e-variable relative to a given alternative
7824:
7475:
5531:
4465:{\displaystyle {\bar {q}}_{W}(Y):=\int _{\Theta _{1}}q_{\theta }(Y)dW(\theta )}
4317:
3906:
2460:
20:
7500:
7149:
Ramdas, Aaditya; Grünwald, Peter; Vovk, Vladimir; Shafer, Glenn (2023-11-01).
7846:
7832:
7765:
7721:
7667:
7625:
7573:
7524:
7516:
7423:
7369:
7278:
7270:
7226:
7203:"Testing by Betting: A Strategy for Statistical and Scientific Communication"
7184:
7114:
7058:
6261:
2473:
67:
7414:
7387:
5528:
e-variables work well in practice but cannot be considered optimal anymore.
2636:, if the null is true, then the probability that a product of e-values will
7783:
7644:"A Logic of Probability, with Application to the Foundations of Statistics"
7431:
7054:
6530:, but this is not a big restriction in practice. In particular, the level-
4313:
1381:
Suppose you can buy a ticket for 1 monetary unit, with nonnegative pay-off
1052:
7544:
Journal of the Royal
Statistical Society Series B: Statistical Methodology
7349:
7249:
Journal of the Royal
Statistical Society Series B: Statistical Methodology
7085:
Journal of the Royal
Statistical Society Series B: Statistical Methodology
7756:
7538:
Candès, Emmanuel; Fan, Yingying; Janson, Lucas; Lv, Jinchi (2018-01-08).
7386:
Wasserman, Larry; Ramdas, Aaditya; Balakrishnan, Sivaraman (2020-07-06).
7361:
7244:
5105:
a specific instantiation of the alternative that explains the data well.
3699:
3424:
Simple alternative, composite null: reverse information projection (RIPr)
7687:
7207:
Journal of the Royal
Statistical Society Series A: Statistics in Society
5596:
which, when applied to a p-variable (a random variable whose value is a
3117:
but this notion has to be suitably modified in the context of e-values.
7713:
7591:
7565:
7218:
7150:
7106:
5669:
One family of admissible calibrators is given by the set of functions
974:
is an e-variable. Conversely, any e-variable relative to a simple null
7617:
7176:
7539:
7202:
3271:
2266:{\displaystyle E:=\prod _{i=1}^{n}E_{i,{\breve {\lambda }}|X^{i-1}}}
7704:
7648:
Journal of the Royal
Statistical Society, Series B (Methodological)
7556:
7404:
7261:
7167:
7097:
2476:. The advantage of e-values in this setting is that they allow for
237:
a fixed sample size or some stopping time. We shall refer to such
7815:
7799:"Asymptotically optimal data analysis for rejecting local realism"
7608:
5597:
1107:{\displaystyle {\mathcal {Q}}=\{Q_{\theta }:\theta \in \Theta \}}
39:
4813:
estimator (such as, for example, the regression coefficients in
3163:(often abbreviated to GRO). In the case of a simple alternative
3147:, used by most authors in the field, is a generalization of the
2466:
50:. For this reason, e-values and their sequential extension, the
4062:
is an e-variable (with the second equality holding if the MLE (
6619:
whose definition is allowed to depend on previous data, i.e.,
5540:. Formally, a calibrator is a nonnegative decreasing function
1059:
e-variables if the null is simple. To see this, note that, if
285:
may also be an unordered bag of outcomes or a single outcome.
1741:
settings. As a prototypical example, consider the case that
7151:"Game-Theoretic Statistics and Safe Anytime-Valid Inference"
6960:
depending on the past, they became dependent on past data).
5532:
A third construction method: p-to-e (and e-to-p) calibration
4312:, originally due to Wald but, in essence, re-discovered by
4230:{\displaystyle H_{1}=\{Q_{\theta }:\theta \in \Theta _{1}\}}
3907:
Simple alternative, composite null: universal inference (UI)
3525:{\displaystyle H_{0}=\{P_{\theta }:\theta \in \Theta _{0}\}}
1174:
as above to be the Bayes marginal distribution with density
7797:
Zhang, Yanbao; Glancy, Scott; Knill, Emanuel (2011-12-22).
7501:"Universal coding, information, prediction, and estimation"
7385:
3633:{\displaystyle E:={\frac {q(Y)}{p_{\curvearrowleft Q}(Y)}}}
7589:
7329:
Grünwald, Peter; De Heide, Rianne; Koolen, Wouter (2024).
6844:{\displaystyle M_{n}=E_{1}\times E_{2}\cdots \times E_{n}}
1241:{\displaystyle q(Y)=\int q_{\theta }(Y)w(\theta )d\theta }
797:
7245:"Estimating means of bounded random variables by betting"
4865:{\displaystyle {\breve {\theta }}\mid X^{0}:=\theta _{0}}
77:
7148:
3420:
has maximal e-power in the sense above, i.e. it is GRO.
6158:
can be written as a (measurable) function of the first
2738:
remains safe (Type-I valid) under optional continuation
2716:, then our Type-I error probability remains bounded by
7328:
3361:
both be simple. Then the likelihood ratio e-variable
2995:) but only for optional continuation (from one sample
2955:. Not all e-variables defined for batches of outcomes
2032:{\displaystyle E_{i,\lambda }:=1+\lambda (X_{i}-\mu )}
7740:"Confidence Sequences for Mean, Variance, and Median"
7688:"E-values: Calibration, combination and applications"
7291:
7001:
6969:
6910:
6877:
6857:
6788:
6742:
6628:
6598:
6571:
6553:
Technically, e-processes are generalizations of test
6536:
6487:
6467:
6447:
6425:
6397:
6377:
6302:
6269:
6216:
6164:
6131:
6085:
6035:
5995:
5933:
5807:
5784:
5726:
5675:
5646:
5626:
5606:
5546:
5500:
5461:
5441:
5283:
5245:
5202:
5166:
5123:
4948:
4921:
4878:
4823:
4789:
4724:
4691:
4649:
4603:
4545:
4480:
4379:
4350:
4330:
4283:
4243:
4178:
4139:
4128:
4107:
4072:
3919:
3816:
3789:
3769:
3735:
3708:
3684:
3648:
3581:
3538:
3473:
3434:
3367:
3328:
3282:
3228:
3208:
3169:
3126:
3081:
3034:
3001:
2981:
2961:
2933:
2909:
2872:
2845:
2818:
2749:
2722:
2694:
2674:
2646:
2610:
2547:
2489:
2431:
2354:
2332:
2282:
2189:
2167:
2134:
2114:
2045:
1978:
1958:
1938:
1918:
1891:
1864:
1832:
1805:
1747:
1714:
1681:
1655:
1625:
1592:
1569:
1549:
1514:
1488:
1462:
1442:
1422:
1387:
1342:
1315:
1256:
1182:
1160:
1140:
1120:
1065:
1030:
1020:
can be written as a likelihood ratio with respect to
980:
960:
896:
874:
854:
808:
780:
742:
715:
693:
660:
556:
521:
489:
469:
425:
376:
341:
306:
271:
243:
223:
196:
138:
118:
91:
6953:{\displaystyle \lambda ={\breve {\lambda }}|X^{i-1}}
5771:{\displaystyle f_{\kappa }(p):=\kappa p^{\kappa -1}}
6727:{\displaystyle P\in H_{0}:{\mathbb {E} }_{P}\leq 1}
6361:{\displaystyle P\in H_{0}:{\mathbb {E} }_{P}\leq 1}
3155:; the optimal e-variable in this sense is known as
1737:form on the other hand is often more convenient in
7348:Wang, Qiuqi; Wang, Ruodu; Ziegel, Johanna (2022).
7335:Journal of the Royal Statistical Society, Series B
7242:
7033:
6987:
6952:
6896:
6863:
6843:
6774:
6726:
6611:
6584:
6542:
6514:
6473:
6453:
6431:
6409:
6383:
6360:
6288:
6248:
6202:
6150:
6117:
6067:
6008:
5977:
5913:
5790:
5770:
5712:
5658:
5632:
5612:
5588:
5515:
5467:
5447:
5417:
5267:
5231:
5188:
5152:
5090:
4934:
4907:
4864:
4801:
4775:
4710:
4677:
4635:
4589:
4528:
4464:
4363:
4336:
4296:
4269:
4229:
4164:
4113:
4093:
4052:
3895:
3802:
3775:
3751:
3721:
3690:
3664:
3632:
3564:
3524:
3459:
3412:
3353:
3314:
3260:
3214:
3194:
3139:
3093:
3059:
3020:
2987:
2967:
2947:
2923:: we may stop the data analysis at any individual
2915:
2891:
2858:
2831:
2793:
2728:
2708:
2680:
2660:
2628:
2591:
2533:
2450:
2417:
2340:
2318:
2265:
2173:
2153:
2120:
2100:
2031:
1964:
1944:
1924:
1904:
1877:
1850:
1818:
1791:
1729:
1696:
1667:
1641:
1611:
1578:
1555:
1535:
1497:
1474:
1448:
1428:
1408:
1365:
1328:
1301:
1240:
1166:
1146:
1126:
1106:
1036:
1012:
966:
944:
880:
860:
840:
786:
762:
728:
699:
674:
644:
540:
507:
475:
431:
408:
360:
327:
277:
249:
229:
209:
182:
124:
104:
19:"E value" redirects here. Not to be confused with
7537:
7297:(Report). Infectious Diseases (except HIV/AIDS).
7243:Waudby-Smith, Ian; Ramdas, Aaditya (2023-02-16).
5778:. Another calibrator is given by integrating out
5713:{\displaystyle \{f_{\kappa }:0<\kappa <1\}}
5108:
3272:Simple alternative, simple null: likelihood ratio
7844:
7796:
6125:be another discrete-time process where for each
5949:
5455:in the formula above by a general stopping time
3944:
7744:Proceedings of the National Academy of Sciences
7392:Proceedings of the National Academy of Sciences
3763:: there exists a specific, unique distribution
2418:{\displaystyle X^{i-1}=(X_{1},\ldots ,X_{i-1})}
7738:Darling, D. A.; Robbins, Herbert (July 1967).
7737:
7592:"Test Martingales, Bayes Factors and p-Values"
7347:
6904:were not dependent on past-data, but by using
4539:To explicate the plug-in method, suppose that
3070:
42:, addressing some shortcomings of the latter.
5232:{\displaystyle {\bar {q}}_{\breve {\theta }}}
5153:{\displaystyle {\bar {q}}_{\breve {\theta }}}
4908:{\displaystyle {\bar {q}}_{\breve {\theta }}}
4809:. In practice one usually takes a "smoothed"
2467:A fundamental property: optional continuation
7081:"False Discovery Rate Control with E-values"
5707:
5676:
4872:. One now recursively constructs a density
4678:{\displaystyle {\breve {\theta }}\mid X^{i}}
4224:
4192:
4159:
4153:
3519:
3487:
3454:
3448:
3348:
3342:
3309:
3296:
3189:
3183:
1101:
1076:
1007:
994:
835:
822:
112:be given as a set of distributions for data
66:and are also related to, yet distinct from,
34:quantify the evidence in the data against a
7079:Wang, Ruodu; Ramdas, Aaditya (2022-07-01).
6441:, "stop as soon as you can reject at level
4776:{\displaystyle X^{i}=(X_{1},\ldots ,X_{i})}
2899:above, and we can therefore even engage in
2866:play the role of 'batches' (full samples)
2319:{\displaystyle {\breve {\lambda }}|X^{i-1}}
1675:a.s. Any e-variable can be written in the
945:{\displaystyle E:={\frac {q(Y)}{p_{0}(Y)}}}
458:
183:{\displaystyle Y=(X_{1},\ldots ,X_{\tau })}
7686:Vovk, Vladimir; Wang, Ruodu (2021-06-01).
7078:
1642:{\displaystyle \lambda \in {\mathbb {R} }}
7814:
7773:
7755:
7703:
7607:
7555:
7413:
7403:
7260:
7166:
7096:
6651:
6325:
4817:), initially set to some "default value"
4643:constitute a stochastic process and let
3232:
1634:
1049:standard, classical likelihood ratio test
380:
7685:
7498:
4237:be composite, such that all elements of
3532:be composite, such that all elements of
2736:. We say that testing based on e-values
2461:Chernoff, Hoeffding and Bernstein bounds
409:{\displaystyle {\mathbb {E} }_{P}\leq 1}
7505:IEEE Transactions on Information Theory
5620:is said to dominate another calibrator
4590:{\displaystyle Y=(X_{1},\ldots ,X_{n})}
2794:{\displaystyle Y_{(1)},Y_{(2)},\ldots }
2592:{\displaystyle Y_{(1)},Y_{(2)},\ldots }
2534:{\displaystyle E_{(1)},E_{(2)},\ldots }
1912:are i.i.d. according to a distribution
1792:{\displaystyle Y=(X_{1},\ldots ,X_{n})}
798:As generalizations of likelihood ratios
7845:
7200:
6075:arriving sequentially, constituting a
5600:), yields an e-variable. A calibrator
4711:{\displaystyle \theta \in \Theta _{1}}
4529:{\displaystyle {\bar {q}}_{W}(Y)/p(Y)}
4094:{\displaystyle {\hat {\theta }}\mid Y}
3222:is simply defined as the expectation
1826:taking values in the bounded interval
368:, its expected value is bounded by 1:
78:Definition and mathematical background
7733:
7731:
7681:
7679:
7677:
7637:
7635:
7585:
7583:
7461:
7196:
7194:
7144:
6203:{\displaystyle (X_{1},\ldots ,X_{n})}
5435:e-process (see below): if we replace
3752:{\displaystyle p_{\curvearrowleft Q}}
3675:Reverse Information Projection (RIPr)
3665:{\displaystyle p_{\curvearrowleft Q}}
1366:{\displaystyle H_{1}:={\mathcal {Q}}}
7641:
7468:Encyclopedia of Statistical Sciences
7451:. J. Wiley & sons, Incorporated.
7444:
7381:
7379:
7324:
7322:
7320:
7318:
7316:
7314:
7312:
7238:
7236:
7142:
7140:
7138:
7136:
7134:
7132:
7130:
7128:
7126:
7124:
3202:, the e-power of a given e-variable
2599:, is itself an e-value, even if the
1114:represents a statistical model, and
7448:Sequential analysis (Section 10.10)
7034:{\displaystyle E_{1},E_{2},\ldots }
6995:The resulting sequence of e-values
6775:{\displaystyle M_{1},M_{2},\ldots }
6515:{\displaystyle E_{n}\geq 1/\alpha }
6249:{\displaystyle E_{1},E_{2},\ldots }
6118:{\displaystyle E_{1},E_{2},\ldots }
6068:{\displaystyle X_{1},X_{2},\ldots }
4636:{\displaystyle X_{1},X_{2},\ldots }
3101:independently of the data we get a
13:
7728:
7674:
7660:10.1111/j.2517-6161.1993.tb01904.x
7632:
7580:
7191:
7041:will then always be an e-process.
5580:
4699:
4417:
4352:
4215:
4129:Composite alternative, simple null
3848:
3791:
3510:
3261:{\displaystyle {\mathbb {E} }_{Q}}
2541:, defined on independent samples
1358:
1141:
1098:
1068:
508:{\displaystyle 0<\alpha \leq 1}
453:
14:
7874:
7376:
7309:
7233:
7121:
5978:{\displaystyle f(t):=\min(1,1/t)}
2629:{\displaystyle 0<\alpha <1}
848:be a simple null hypothesis. Let
6077:discrete-time stochastic process
2803:Doob's optional stopping theorem
954:be their likelihood ratio. Then
7790:
7531:
7492:
7455:
6560:
6296:is an e-variable, i.e. for all
6289:{\displaystyle \tau ,E_{\tau }}
5429:classical likelihood ratio test
4270:{\displaystyle H_{0}\cup H_{1}}
3565:{\displaystyle H_{0}\cup H_{1}}
3413:{\displaystyle E=q(Y)/p_{0}(Y)}
3315:{\displaystyle H_{0}=\{P_{0}\}}
1302:{\displaystyle E=q(Y)/p_{0}(Y)}
1013:{\displaystyle H_{0}=\{P_{0}\}}
841:{\displaystyle H_{0}=\{P_{0}\}}
763:{\displaystyle 1/E\leq \alpha }
7853:Statistical hypothesis testing
7438:
7341:
7285:
7072:
6930:
6897:{\displaystyle E_{i,\lambda }}
6715:
6676:
6662:
6349:
6336:
6197:
6165:
6019:
5972:
5952:
5943:
5937:
5899:
5883:
5743:
5737:
5589:{\displaystyle f:\rightarrow }
5583:
5571:
5568:
5565:
5553:
5409:
5396:
5375:
5359:
5346:
5268:{\displaystyle {\bar {q}}_{W}}
5253:
5210:
5189:{\displaystyle {\bar {q}}_{W}}
5174:
5131:
5109:Composite null and alternative
5085:
5053:
4988:
4975:
4956:
4886:
4770:
4738:
4584:
4552:
4523:
4517:
4506:
4500:
4488:
4459:
4453:
4444:
4438:
4405:
4399:
4387:
4316:as "prequential plug-in" and
4079:
4039:
4033:
4019:
4001:
3995:
3975:
3969:
3938:
3932:
3890:
3884:
3875:
3869:
3836:
3830:
3624:
3618:
3600:
3594:
3407:
3401:
3383:
3377:
3255:
3243:
3052:
3040:
3013:
3007:
2884:
2878:
2780:
2774:
2761:
2755:
2578:
2572:
2559:
2553:
2520:
2514:
2501:
2495:
2451:{\displaystyle E_{i,\lambda }}
2412:
2374:
2296:
2241:
2154:{\displaystyle E_{i,\lambda }}
2095:
2078:
2066:
2052:
2026:
2007:
1845:
1833:
1786:
1754:
1536:{\displaystyle E:=1+\lambda U}
1403:
1397:
1296:
1290:
1272:
1266:
1229:
1223:
1217:
1211:
1192:
1186:
936:
930:
915:
909:
630:
624:
612:
592:
419:The value taken by e-variable
397:
391:
322:
316:
177:
145:
28:statistical hypothesis testing
1:
7065:
6461:-level, i.e. at the smallest
6024:
1952:; no other assumptions about
868:be any other distribution on
7642:Vovk, V. G. (January 1993).
7201:Shafer, Glenn (2021-04-01).
4064:maximum likelihood estimator
2903:"within" the original batch
2101:{\displaystyle \lambda \in }
7:
7462:Dawid, A. P. (2004-07-15).
7303:10.1101/2022.12.15.22283474
6988:{\displaystyle 1,2,\ldots }
5516:{\displaystyle 1+\lambda U}
4364:{\displaystyle \Theta _{1}}
4165:{\displaystyle H_{0}=\{P\}}
3803:{\displaystyle \Theta _{0}}
3460:{\displaystyle H_{1}=\{Q\}}
3354:{\displaystyle H_{1}=\{Q\}}
3195:{\displaystyle H_{1}=\{Q\}}
3071:Construction and optimality
1730:{\displaystyle 1+\lambda U}
1697:{\displaystyle 1+\lambda U}
10:
7879:
7825:10.1103/physreva.84.062118
7499:Rissanen, J. (July 1984).
7476:10.1002/0471667196.ess0335
7044:
2348:, based only on past data
2341:{\displaystyle {\lambda }}
1612:{\displaystyle P\in H_{0}}
1376:
1309:is also a Bayes factor of
541:{\displaystyle P\in H_{0}}
361:{\displaystyle P\in H_{0}}
18:
3060:{\displaystyle Y_{(j+1)}}
2948:{\displaystyle 1/\alpha }
2709:{\displaystyle 1/\alpha }
2661:{\displaystyle 1/\alpha }
7692:The Annals of Statistics
7517:10.1109/tit.1984.1056936
6864:{\displaystyle \lambda }
4123:universal inference (UI)
2425:, and designed to make
2121:{\displaystyle \lambda }
1482:, which has expectation
684:e-value based test with
459:As conservative p-values
7415:10.1073/pnas.1922664117
7354:SSRN Electronic Journal
6543:{\displaystyle \alpha }
6454:{\displaystyle \alpha }
6432:{\displaystyle \alpha }
6151:{\displaystyle n,E_{n}}
5791:{\displaystyle \kappa }
5659:{\displaystyle f\geq g}
4802:{\displaystyle i\geq 0}
4474:and use the e-variable
3021:{\displaystyle Y_{(j)}}
2892:{\displaystyle Y_{(j)}}
2729:{\displaystyle \alpha }
2681:{\displaystyle \alpha }
1668:{\displaystyle E\geq 0}
1147:{\displaystyle \Theta }
787:{\displaystyle \alpha }
700:{\displaystyle \alpha }
7464:"Prequential Analysis"
7445:Wald, Abraham (1947).
7271:10.1093/jrsssb/qkad009
7035:
6989:
6954:
6898:
6865:
6845:
6776:
6728:
6613:
6586:
6544:
6516:
6475:
6455:
6433:
6411:
6385:
6362:
6290:
6250:
6204:
6152:
6119:
6069:
6010:
5979:
5915:
5792:
5772:
5714:
5660:
5634:
5614:
5590:
5517:
5469:
5449:
5419:
5307:
5269:
5233:
5190:
5154:
5092:
5014:
4936:
4909:
4866:
4803:
4777:
4712:
4679:
4637:
4591:
4530:
4466:
4365:
4338:
4298:
4271:
4231:
4166:
4115:
4095:
4054:
3897:
3804:
3777:
3753:
3723:
3692:
3666:
3634:
3566:
3526:
3461:
3414:
3355:
3316:
3262:
3216:
3196:
3141:
3095:
3061:
3022:
2989:
2969:
2949:
2917:
2893:
2860:
2833:
2795:
2730:
2710:
2682:
2662:
2630:
2593:
2535:
2478:optional continuation.
2452:
2419:
2342:
2320:
2267:
2216:
2181:by taking the product
2175:
2155:
2122:
2102:
2033:
1966:
1946:
1926:
1906:
1879:
1852:
1820:
1793:
1731:
1698:
1669:
1643:
1613:
1580:
1579:{\displaystyle \leq 0}
1557:
1537:
1499:
1498:{\displaystyle \leq 0}
1476:
1450:
1430:
1410:
1409:{\displaystyle E=E(Y)}
1367:
1330:
1303:
1242:
1168:
1148:
1128:
1108:
1038:
1014:
968:
946:
882:
862:
842:
788:
764:
730:
701:
682:is a p-value, and the
676:
646:
542:
509:
477:
446:In practice, the term
433:
410:
362:
329:
328:{\displaystyle E=E(Y)}
279:
251:
231:
211:
184:
126:
106:
7388:"Universal inference"
7036:
6990:
6955:
6899:
6866:
6846:
6777:
6729:
6614:
6612:{\displaystyle X_{i}}
6587:
6585:{\displaystyle S_{i}}
6545:
6517:
6476:
6456:
6434:
6412:
6410:{\displaystyle n=100}
6386:
6363:
6291:
6251:
6205:
6153:
6120:
6070:
6011:
6009:{\displaystyle H_{1}}
5980:
5916:
5793:
5773:
5715:
5661:
5635:
5615:
5591:
5518:
5470:
5468:{\displaystyle \tau }
5450:
5420:
5287:
5270:
5234:
5191:
5155:
5093:
4994:
4937:
4935:{\displaystyle X^{n}}
4910:
4867:
4804:
4778:
4713:
4680:
4638:
4592:
4531:
4467:
4366:
4339:
4299:
4297:{\displaystyle H_{1}}
4272:
4232:
4167:
4116:
4096:
4055:
3898:
3805:
3778:
3754:
3724:
3722:{\displaystyle H_{0}}
3693:
3667:
3635:
3567:
3527:
3462:
3415:
3356:
3317:
3263:
3217:
3197:
3142:
3140:{\displaystyle H_{1}}
3096:
3062:
3023:
2990:
2970:
2950:
2918:
2894:
2861:
2859:{\displaystyle X_{i}}
2834:
2832:{\displaystyle X_{i}}
2796:
2731:
2711:
2683:
2663:
2631:
2594:
2536:
2453:
2420:
2343:
2321:
2268:
2196:
2176:
2156:
2123:
2103:
2034:
1967:
1947:
1927:
1907:
1905:{\displaystyle X_{i}}
1880:
1878:{\displaystyle H_{0}}
1853:
1821:
1819:{\displaystyle X_{i}}
1794:
1732:
1699:
1670:
1644:
1614:
1581:
1558:
1538:
1500:
1477:
1451:
1431:
1411:
1368:
1331:
1329:{\displaystyle H_{0}}
1304:
1243:
1169:
1149:
1129:
1109:
1039:
1015:
969:
947:
883:
863:
843:
789:
765:
731:
729:{\displaystyle P_{0}}
702:
677:
647:
543:
510:
478:
434:
411:
363:
330:
280:
252:
232:
230:{\displaystyle \tau }
217:a single outcome and
212:
210:{\displaystyle X_{i}}
185:
127:
107:
105:{\displaystyle H_{0}}
7858:Statistical concepts
7757:10.1073/pnas.58.1.66
7362:10.2139/ssrn.4206997
6999:
6967:
6908:
6875:
6855:
6786:
6740:
6626:
6596:
6569:
6534:
6485:
6465:
6445:
6423:
6395:
6375:
6300:
6267:
6214:
6162:
6129:
6083:
6033:
5993:
5931:
5805:
5782:
5724:
5673:
5644:
5624:
5604:
5544:
5498:
5459:
5439:
5281:
5243:
5200:
5164:
5121:
4946:
4919:
4876:
4821:
4787:
4722:
4689:
4685:be an estimator of
4647:
4601:
4543:
4478:
4377:
4348:
4328:
4281:
4241:
4176:
4137:
4105:
4070:
3917:
3814:
3787:
3767:
3759:is given by a Bayes
3733:
3706:
3682:
3646:
3579:
3536:
3471:
3432:
3365:
3326:
3280:
3226:
3206:
3167:
3124:
3094:{\displaystyle E:=1}
3079:
3032:
2999:
2979:
2959:
2931:
2907:
2870:
2843:
2816:
2747:
2720:
2692:
2672:
2644:
2608:
2545:
2487:
2429:
2352:
2330:
2280:
2187:
2165:
2132:
2112:
2043:
1976:
1956:
1945:{\displaystyle \mu }
1936:
1916:
1889:
1862:
1830:
1803:
1745:
1712:
1679:
1653:
1623:
1590:
1567:
1547:
1512:
1486:
1460:
1440:
1420:
1385:
1340:
1313:
1254:
1180:
1158:
1138:
1118:
1063:
1028:
978:
958:
894:
872:
852:
806:
778:
740:
713:
691:
658:
554:
519:
487:
467:
423:
374:
339:
335:such that under all
304:
269:
241:
221:
194:
136:
116:
89:
48:Type-I error control
7596:Statistical Science
7398:(29): 16880–16890.
7155:Statistical Science
6524:sequential analysis
5822:
3161:growth-rate optimal
2640:become larger than
2326:is an estimate for
1704:form although with
1649:is chosen so that
1475:{\displaystyle E-1}
1134:a prior density on
675:{\displaystyle 1/E}
463:For any e-variable
16:Statistical concept
7863:Probability theory
7714:10.1214/20-aos2020
7566:10.1111/rssb.12265
7219:10.1111/rssa.12647
7107:10.1111/rssb.12489
7031:
6985:
6950:
6894:
6861:
6841:
6772:
6724:
6609:
6582:
6540:
6512:
6471:
6451:
6429:
6407:
6381:
6358:
6286:
6246:
6210:outcomes. We call
6200:
6148:
6115:
6065:
6029:Now consider data
6006:
5975:
5911:
5808:
5788:
5768:
5710:
5656:
5630:
5610:
5586:
5538:p-to-e calibrators
5513:
5485:model-X assumption
5465:
5445:
5415:
5265:
5229:
5186:
5150:
5088:
4932:
4905:
4862:
4811:maximum likelihood
4799:
4773:
4708:
4675:
4633:
4587:
4526:
4462:
4361:
4334:
4306:method of mixtures
4294:
4267:
4227:
4162:
4111:
4091:
4050:
3965:
3893:
3800:
3773:
3749:
3719:
3688:
3662:
3630:
3562:
3522:
3457:
3410:
3351:
3312:
3258:
3212:
3192:
3137:
3091:
3057:
3018:
2985:
2965:
2945:
2913:
2889:
2856:
2829:
2807:Ville's inequality
2791:
2726:
2706:
2678:
2658:
2626:
2589:
2531:
2448:
2415:
2338:
2316:
2263:
2171:
2151:
2118:
2098:
2029:
1962:
1942:
1922:
1902:
1875:
1848:
1816:
1789:
1727:
1694:
1665:
1639:
1609:
1576:
1553:
1533:
1495:
1472:
1446:
1426:
1416:. The statements "
1406:
1363:
1326:
1299:
1238:
1164:
1154:, then we can set
1144:
1124:
1104:
1034:
1010:
964:
942:
878:
858:
838:
784:
760:
726:
697:
686:significance level
672:
642:
538:
505:
473:
429:
406:
358:
325:
275:
265:But in some cases
263:batch of outcomes.
247:
227:
207:
180:
122:
102:
7803:Physical Review A
7618:10.1214/10-sts347
7485:978-0-471-15044-2
7177:10.1214/23-sts894
6926:
6474:{\displaystyle n}
6384:{\displaystyle n}
5909:
5633:{\displaystyle g}
5613:{\displaystyle f}
5448:{\displaystyle n}
5413:
5378:
5322:
5256:
5225:
5213:
5177:
5146:
5134:
5029:
4971:
4959:
4901:
4889:
4833:
4659:
4491:
4390:
4337:{\displaystyle W}
4114:{\displaystyle Y}
4082:
4043:
4022:
3979:
3943:
3776:{\displaystyle W}
3691:{\displaystyle Q}
3628:
3215:{\displaystyle S}
3114:statistical power
2988:{\displaystyle Y}
2968:{\displaystyle Y}
2916:{\displaystyle Y}
2901:optional stopping
2292:
2237:
2174:{\displaystyle Y}
1965:{\displaystyle P}
1925:{\displaystyle P}
1563:has expectation
1556:{\displaystyle U}
1449:{\displaystyle E}
1429:{\displaystyle E}
1167:{\displaystyle Q}
1127:{\displaystyle w}
1037:{\displaystyle Q}
967:{\displaystyle E}
940:
881:{\displaystyle Y}
861:{\displaystyle Q}
638:
634:
617:
579:
476:{\displaystyle E}
432:{\displaystyle E}
278:{\displaystyle Y}
250:{\displaystyle Y}
125:{\displaystyle Y}
64:likelihood ratios
7870:
7837:
7836:
7818:
7794:
7788:
7787:
7777:
7759:
7735:
7726:
7725:
7707:
7683:
7672:
7671:
7639:
7630:
7629:
7611:
7587:
7578:
7577:
7559:
7535:
7529:
7528:
7496:
7490:
7489:
7459:
7453:
7452:
7442:
7436:
7435:
7417:
7407:
7383:
7374:
7373:
7345:
7339:
7338:
7326:
7307:
7306:
7289:
7283:
7282:
7264:
7240:
7231:
7230:
7198:
7189:
7188:
7170:
7146:
7119:
7118:
7100:
7076:
7040:
7038:
7037:
7032:
7024:
7023:
7011:
7010:
6994:
6992:
6991:
6986:
6959:
6957:
6956:
6951:
6949:
6948:
6933:
6928:
6927:
6919:
6903:
6901:
6900:
6895:
6893:
6892:
6870:
6868:
6867:
6862:
6850:
6848:
6847:
6842:
6840:
6839:
6824:
6823:
6811:
6810:
6798:
6797:
6781:
6779:
6778:
6773:
6765:
6764:
6752:
6751:
6733:
6731:
6730:
6725:
6714:
6713:
6689:
6688:
6679:
6674:
6673:
6661:
6660:
6655:
6654:
6644:
6643:
6618:
6616:
6615:
6610:
6608:
6607:
6591:
6589:
6588:
6583:
6581:
6580:
6555:supermartingales
6549:
6547:
6546:
6541:
6521:
6519:
6518:
6513:
6508:
6497:
6496:
6480:
6478:
6477:
6472:
6460:
6458:
6457:
6452:
6439:-aggressive rule
6438:
6436:
6435:
6430:
6416:
6414:
6413:
6408:
6390:
6388:
6387:
6382:
6367:
6365:
6364:
6359:
6348:
6347:
6335:
6334:
6329:
6328:
6318:
6317:
6295:
6293:
6292:
6287:
6285:
6284:
6255:
6253:
6252:
6247:
6239:
6238:
6226:
6225:
6209:
6207:
6206:
6201:
6196:
6195:
6177:
6176:
6157:
6155:
6154:
6149:
6147:
6146:
6124:
6122:
6121:
6116:
6108:
6107:
6095:
6094:
6074:
6072:
6071:
6066:
6058:
6057:
6045:
6044:
6015:
6013:
6012:
6007:
6005:
6004:
5984:
5982:
5981:
5976:
5968:
5920:
5918:
5917:
5912:
5910:
5908:
5907:
5906:
5878:
5852:
5841:
5840:
5821:
5816:
5797:
5795:
5794:
5789:
5777:
5775:
5774:
5769:
5767:
5766:
5736:
5735:
5719:
5717:
5716:
5711:
5688:
5687:
5665:
5663:
5662:
5657:
5639:
5637:
5636:
5631:
5619:
5617:
5616:
5611:
5595:
5593:
5592:
5587:
5522:
5520:
5519:
5514:
5474:
5472:
5471:
5466:
5454:
5452:
5451:
5446:
5424:
5422:
5421:
5416:
5414:
5412:
5408:
5407:
5395:
5394:
5393:
5392:
5380:
5379:
5371:
5362:
5358:
5357:
5345:
5344:
5343:
5342:
5324:
5323:
5315:
5306:
5301:
5285:
5274:
5272:
5271:
5266:
5264:
5263:
5258:
5257:
5249:
5238:
5236:
5235:
5230:
5228:
5227:
5226:
5218:
5215:
5214:
5206:
5195:
5193:
5192:
5187:
5185:
5184:
5179:
5178:
5170:
5159:
5157:
5156:
5151:
5149:
5148:
5147:
5139:
5136:
5135:
5127:
5097:
5095:
5094:
5089:
5084:
5083:
5065:
5064:
5052:
5051:
5050:
5049:
5031:
5030:
5022:
5013:
5008:
4987:
4986:
4974:
4973:
4972:
4964:
4961:
4960:
4952:
4941:
4939:
4938:
4933:
4931:
4930:
4914:
4912:
4911:
4906:
4904:
4903:
4902:
4894:
4891:
4890:
4882:
4871:
4869:
4868:
4863:
4861:
4860:
4848:
4847:
4835:
4834:
4826:
4815:ridge regression
4808:
4806:
4805:
4800:
4782:
4780:
4779:
4774:
4769:
4768:
4750:
4749:
4734:
4733:
4717:
4715:
4714:
4709:
4707:
4706:
4684:
4682:
4681:
4676:
4674:
4673:
4661:
4660:
4652:
4642:
4640:
4639:
4634:
4626:
4625:
4613:
4612:
4596:
4594:
4593:
4588:
4583:
4582:
4564:
4563:
4535:
4533:
4532:
4527:
4513:
4499:
4498:
4493:
4492:
4484:
4471:
4469:
4468:
4463:
4437:
4436:
4427:
4426:
4425:
4424:
4398:
4397:
4392:
4391:
4383:
4370:
4368:
4367:
4362:
4360:
4359:
4343:
4341:
4340:
4335:
4303:
4301:
4300:
4295:
4293:
4292:
4276:
4274:
4273:
4268:
4266:
4265:
4253:
4252:
4236:
4234:
4233:
4228:
4223:
4222:
4204:
4203:
4188:
4187:
4171:
4169:
4168:
4163:
4149:
4148:
4120:
4118:
4117:
4112:
4100:
4098:
4097:
4092:
4084:
4083:
4075:
4059:
4057:
4056:
4051:
4049:
4045:
4044:
4042:
4032:
4031:
4024:
4023:
4015:
4011:
4004:
3990:
3980:
3978:
3964:
3963:
3962:
3941:
3927:
3902:
3900:
3899:
3894:
3868:
3867:
3858:
3857:
3856:
3855:
3829:
3828:
3809:
3807:
3806:
3801:
3799:
3798:
3782:
3780:
3779:
3774:
3761:marginal density
3758:
3756:
3755:
3750:
3748:
3747:
3728:
3726:
3725:
3720:
3718:
3717:
3697:
3695:
3694:
3689:
3671:
3669:
3668:
3663:
3661:
3660:
3639:
3637:
3636:
3631:
3629:
3627:
3617:
3616:
3603:
3589:
3571:
3569:
3568:
3563:
3561:
3560:
3548:
3547:
3531:
3529:
3528:
3523:
3518:
3517:
3499:
3498:
3483:
3482:
3466:
3464:
3463:
3458:
3444:
3443:
3419:
3417:
3416:
3411:
3400:
3399:
3390:
3360:
3358:
3357:
3352:
3338:
3337:
3321:
3319:
3318:
3313:
3308:
3307:
3292:
3291:
3267:
3265:
3264:
3259:
3242:
3241:
3236:
3235:
3221:
3219:
3218:
3213:
3201:
3199:
3198:
3193:
3179:
3178:
3146:
3144:
3143:
3138:
3136:
3135:
3100:
3098:
3097:
3092:
3066:
3064:
3063:
3058:
3056:
3055:
3027:
3025:
3024:
3019:
3017:
3016:
2994:
2992:
2991:
2986:
2974:
2972:
2971:
2966:
2954:
2952:
2951:
2946:
2941:
2922:
2920:
2919:
2914:
2898:
2896:
2895:
2890:
2888:
2887:
2865:
2863:
2862:
2857:
2855:
2854:
2838:
2836:
2835:
2830:
2828:
2827:
2800:
2798:
2797:
2792:
2784:
2783:
2765:
2764:
2735:
2733:
2732:
2727:
2715:
2713:
2712:
2707:
2702:
2687:
2685:
2684:
2679:
2667:
2665:
2664:
2659:
2654:
2635:
2633:
2632:
2627:
2598:
2596:
2595:
2590:
2582:
2581:
2563:
2562:
2540:
2538:
2537:
2532:
2524:
2523:
2505:
2504:
2457:
2455:
2454:
2449:
2447:
2446:
2424:
2422:
2421:
2416:
2411:
2410:
2386:
2385:
2370:
2369:
2347:
2345:
2344:
2339:
2337:
2325:
2323:
2322:
2317:
2315:
2314:
2299:
2294:
2293:
2285:
2272:
2270:
2269:
2264:
2262:
2261:
2260:
2259:
2244:
2239:
2238:
2230:
2215:
2210:
2180:
2178:
2177:
2172:
2160:
2158:
2157:
2152:
2150:
2149:
2127:
2125:
2124:
2119:
2108:(these are the
2107:
2105:
2104:
2099:
2091:
2065:
2038:
2036:
2035:
2030:
2019:
2018:
1994:
1993:
1971:
1969:
1968:
1963:
1951:
1949:
1948:
1943:
1931:
1929:
1928:
1923:
1911:
1909:
1908:
1903:
1901:
1900:
1884:
1882:
1881:
1876:
1874:
1873:
1857:
1855:
1854:
1851:{\displaystyle }
1849:
1825:
1823:
1822:
1817:
1815:
1814:
1798:
1796:
1795:
1790:
1785:
1784:
1766:
1765:
1736:
1734:
1733:
1728:
1703:
1701:
1700:
1695:
1674:
1672:
1671:
1666:
1648:
1646:
1645:
1640:
1638:
1637:
1618:
1616:
1615:
1610:
1608:
1607:
1585:
1583:
1582:
1577:
1562:
1560:
1559:
1554:
1542:
1540:
1539:
1534:
1504:
1502:
1501:
1496:
1481:
1479:
1478:
1473:
1455:
1453:
1452:
1447:
1435:
1433:
1432:
1427:
1415:
1413:
1412:
1407:
1372:
1370:
1369:
1364:
1362:
1361:
1352:
1351:
1335:
1333:
1332:
1327:
1325:
1324:
1308:
1306:
1305:
1300:
1289:
1288:
1279:
1247:
1245:
1244:
1239:
1210:
1209:
1173:
1171:
1170:
1165:
1153:
1151:
1150:
1145:
1133:
1131:
1130:
1125:
1113:
1111:
1110:
1105:
1088:
1087:
1072:
1071:
1043:
1041:
1040:
1035:
1019:
1017:
1016:
1011:
1006:
1005:
990:
989:
973:
971:
970:
965:
951:
949:
948:
943:
941:
939:
929:
928:
918:
904:
887:
885:
884:
879:
867:
865:
864:
859:
847:
845:
844:
839:
834:
833:
818:
817:
793:
791:
790:
785:
769:
767:
766:
761:
750:
735:
733:
732:
727:
725:
724:
706:
704:
703:
698:
681:
679:
678:
673:
668:
651:
649:
648:
643:
636:
635:
633:
619:
615:
602:
585:
581:
580:
572:
548:, it holds that
547:
545:
544:
539:
537:
536:
514:
512:
511:
506:
482:
480:
479:
474:
438:
436:
435:
430:
415:
413:
412:
407:
390:
389:
384:
383:
367:
365:
364:
359:
357:
356:
334:
332:
331:
326:
300:random variable
284:
282:
281:
276:
256:
254:
253:
248:
236:
234:
233:
228:
216:
214:
213:
208:
206:
205:
189:
187:
186:
181:
176:
175:
157:
156:
131:
129:
128:
123:
111:
109:
108:
103:
101:
100:
72:supermartingales
56:multiple testing
7878:
7877:
7873:
7872:
7871:
7869:
7868:
7867:
7843:
7842:
7841:
7840:
7795:
7791:
7736:
7729:
7684:
7675:
7640:
7633:
7588:
7581:
7536:
7532:
7497:
7493:
7486:
7460:
7456:
7443:
7439:
7384:
7377:
7350:"E-backtesting"
7346:
7342:
7327:
7310:
7290:
7286:
7241:
7234:
7199:
7192:
7147:
7122:
7077:
7073:
7068:
7051:Herbert Robbins
7047:
7019:
7015:
7006:
7002:
7000:
6997:
6996:
6968:
6965:
6964:
6938:
6934:
6929:
6918:
6917:
6909:
6906:
6905:
6882:
6878:
6876:
6873:
6872:
6871:, the e-values
6856:
6853:
6852:
6835:
6831:
6819:
6815:
6806:
6802:
6793:
6789:
6787:
6784:
6783:
6760:
6756:
6747:
6743:
6741:
6738:
6737:
6703:
6699:
6684:
6680:
6675:
6669:
6665:
6656:
6650:
6649:
6648:
6639:
6635:
6627:
6624:
6623:
6603:
6599:
6597:
6594:
6593:
6576:
6572:
6570:
6567:
6566:
6563:
6535:
6532:
6531:
6504:
6492:
6488:
6486:
6483:
6482:
6466:
6463:
6462:
6446:
6443:
6442:
6424:
6421:
6420:
6396:
6393:
6392:
6376:
6373:
6372:
6343:
6339:
6330:
6324:
6323:
6322:
6313:
6309:
6301:
6298:
6297:
6280:
6276:
6268:
6265:
6264:
6234:
6230:
6221:
6217:
6215:
6212:
6211:
6191:
6187:
6172:
6168:
6163:
6160:
6159:
6142:
6138:
6130:
6127:
6126:
6103:
6099:
6090:
6086:
6084:
6081:
6080:
6053:
6049:
6040:
6036:
6034:
6031:
6030:
6027:
6022:
6000:
5996:
5994:
5991:
5990:
5964:
5932:
5929:
5928:
5902:
5898:
5879:
5853:
5851:
5830:
5826:
5817:
5812:
5806:
5803:
5802:
5783:
5780:
5779:
5756:
5752:
5731:
5727:
5725:
5722:
5721:
5683:
5679:
5674:
5671:
5670:
5645:
5642:
5641:
5625:
5622:
5621:
5605:
5602:
5601:
5545:
5542:
5541:
5534:
5499:
5496:
5495:
5460:
5457:
5456:
5440:
5437:
5436:
5403:
5399:
5388:
5384:
5370:
5369:
5368:
5364:
5363:
5353:
5349:
5332:
5328:
5314:
5313:
5312:
5308:
5302:
5291:
5286:
5284:
5282:
5279:
5278:
5259:
5248:
5247:
5246:
5244:
5241:
5240:
5217:
5216:
5205:
5204:
5203:
5201:
5198:
5197:
5180:
5169:
5168:
5167:
5165:
5162:
5161:
5138:
5137:
5126:
5125:
5124:
5122:
5119:
5118:
5111:
5073:
5069:
5060:
5056:
5039:
5035:
5021:
5020:
5019:
5015:
5009:
4998:
4982:
4978:
4963:
4962:
4951:
4950:
4949:
4947:
4944:
4943:
4926:
4922:
4920:
4917:
4916:
4893:
4892:
4881:
4880:
4879:
4877:
4874:
4873:
4856:
4852:
4843:
4839:
4825:
4824:
4822:
4819:
4818:
4788:
4785:
4784:
4764:
4760:
4745:
4741:
4729:
4725:
4723:
4720:
4719:
4718:based on data
4702:
4698:
4690:
4687:
4686:
4669:
4665:
4651:
4650:
4648:
4645:
4644:
4621:
4617:
4608:
4604:
4602:
4599:
4598:
4578:
4574:
4559:
4555:
4544:
4541:
4540:
4509:
4494:
4483:
4482:
4481:
4479:
4476:
4475:
4432:
4428:
4420:
4416:
4415:
4411:
4393:
4382:
4381:
4380:
4378:
4375:
4374:
4355:
4351:
4349:
4346:
4345:
4329:
4326:
4325:
4320:as "predictive
4288:
4284:
4282:
4279:
4278:
4261:
4257:
4248:
4244:
4242:
4239:
4238:
4218:
4214:
4199:
4195:
4183:
4179:
4177:
4174:
4173:
4172:be simple and
4144:
4140:
4138:
4135:
4134:
4131:
4106:
4103:
4102:
4074:
4073:
4071:
4068:
4067:
4014:
4013:
4012:
4007:
4006:
4005:
3991:
3989:
3985:
3981:
3958:
3954:
3947:
3942:
3928:
3926:
3918:
3915:
3914:
3909:
3863:
3859:
3851:
3847:
3846:
3842:
3821:
3817:
3815:
3812:
3811:
3794:
3790:
3788:
3785:
3784:
3768:
3765:
3764:
3740:
3736:
3734:
3731:
3730:
3713:
3709:
3707:
3704:
3703:
3683:
3680:
3679:
3653:
3649:
3647:
3644:
3643:
3609:
3605:
3604:
3590:
3588:
3580:
3577:
3576:
3556:
3552:
3543:
3539:
3537:
3534:
3533:
3513:
3509:
3494:
3490:
3478:
3474:
3472:
3469:
3468:
3439:
3435:
3433:
3430:
3429:
3426:
3395:
3391:
3386:
3366:
3363:
3362:
3333:
3329:
3327:
3324:
3323:
3303:
3299:
3287:
3283:
3281:
3278:
3277:
3274:
3237:
3231:
3230:
3229:
3227:
3224:
3223:
3207:
3204:
3203:
3174:
3170:
3168:
3165:
3164:
3149:Kelly criterion
3131:
3127:
3125:
3122:
3121:
3080:
3077:
3076:
3073:
3039:
3035:
3033:
3030:
3029:
3006:
3002:
3000:
2997:
2996:
2980:
2977:
2976:
2960:
2957:
2956:
2937:
2932:
2929:
2928:
2908:
2905:
2904:
2877:
2873:
2871:
2868:
2867:
2850:
2846:
2844:
2841:
2840:
2823:
2819:
2817:
2814:
2813:
2773:
2769:
2754:
2750:
2748:
2745:
2744:
2721:
2718:
2717:
2698:
2693:
2690:
2689:
2673:
2670:
2669:
2650:
2645:
2642:
2641:
2609:
2606:
2605:
2571:
2567:
2552:
2548:
2546:
2543:
2542:
2513:
2509:
2494:
2490:
2488:
2485:
2484:
2469:
2436:
2432:
2430:
2427:
2426:
2400:
2396:
2381:
2377:
2359:
2355:
2353:
2350:
2349:
2333:
2331:
2328:
2327:
2304:
2300:
2295:
2284:
2283:
2281:
2278:
2277:
2249:
2245:
2240:
2229:
2228:
2221:
2217:
2211:
2200:
2188:
2185:
2184:
2166:
2163:
2162:
2139:
2135:
2133:
2130:
2129:
2113:
2110:
2109:
2087:
2061:
2044:
2041:
2040:
2014:
2010:
1983:
1979:
1977:
1974:
1973:
1957:
1954:
1953:
1937:
1934:
1933:
1917:
1914:
1913:
1896:
1892:
1890:
1887:
1886:
1869:
1865:
1863:
1860:
1859:
1858:. According to
1831:
1828:
1827:
1810:
1806:
1804:
1801:
1800:
1780:
1776:
1761:
1757:
1746:
1743:
1742:
1713:
1710:
1709:
1680:
1677:
1676:
1654:
1651:
1650:
1633:
1632:
1624:
1621:
1620:
1603:
1599:
1591:
1588:
1587:
1568:
1565:
1564:
1548:
1545:
1544:
1513:
1510:
1509:
1487:
1484:
1483:
1461:
1458:
1457:
1441:
1438:
1437:
1421:
1418:
1417:
1386:
1383:
1382:
1379:
1357:
1356:
1347:
1343:
1341:
1338:
1337:
1320:
1316:
1314:
1311:
1310:
1284:
1280:
1275:
1255:
1252:
1251:
1205:
1201:
1181:
1178:
1177:
1159:
1156:
1155:
1139:
1136:
1135:
1119:
1116:
1115:
1083:
1079:
1067:
1066:
1064:
1061:
1060:
1029:
1026:
1025:
1001:
997:
985:
981:
979:
976:
975:
959:
956:
955:
924:
920:
919:
905:
903:
895:
892:
891:
873:
870:
869:
853:
850:
849:
829:
825:
813:
809:
807:
804:
803:
800:
779:
776:
775:
746:
741:
738:
737:
720:
716:
714:
711:
710:
692:
689:
688:
664:
659:
656:
655:
623:
618:
598:
571:
564:
560:
555:
552:
551:
532:
528:
520:
517:
516:
488:
485:
484:
468:
465:
464:
461:
456:
454:Interpretations
424:
421:
420:
385:
379:
378:
377:
375:
372:
371:
352:
348:
340:
337:
336:
305:
302:
301:
270:
267:
266:
242:
239:
238:
222:
219:
218:
201:
197:
195:
192:
191:
171:
167:
152:
148:
137:
134:
133:
117:
114:
113:
96:
92:
90:
87:
86:
84:null hypothesis
80:
36:null hypothesis
24:
17:
12:
11:
5:
7876:
7866:
7865:
7860:
7855:
7839:
7838:
7789:
7727:
7673:
7654:(2): 317–341.
7631:
7579:
7550:(3): 551–577.
7530:
7511:(4): 629–636.
7491:
7484:
7454:
7437:
7375:
7340:
7331:"Safe Testing"
7308:
7284:
7232:
7213:(2): 407–431.
7190:
7120:
7091:(3): 822–852.
7070:
7069:
7067:
7064:
7046:
7043:
7030:
7027:
7022:
7018:
7014:
7009:
7005:
6984:
6981:
6978:
6975:
6972:
6947:
6944:
6941:
6937:
6932:
6925:
6922:
6916:
6913:
6891:
6888:
6885:
6881:
6860:
6838:
6834:
6830:
6827:
6822:
6818:
6814:
6809:
6805:
6801:
6796:
6792:
6771:
6768:
6763:
6759:
6755:
6750:
6746:
6723:
6720:
6717:
6712:
6709:
6706:
6702:
6698:
6695:
6692:
6687:
6683:
6678:
6672:
6668:
6664:
6659:
6653:
6647:
6642:
6638:
6634:
6631:
6606:
6602:
6579:
6575:
6562:
6559:
6539:
6511:
6507:
6503:
6500:
6495:
6491:
6470:
6450:
6428:
6406:
6403:
6400:
6380:
6357:
6354:
6351:
6346:
6342:
6338:
6333:
6327:
6321:
6316:
6312:
6308:
6305:
6283:
6279:
6275:
6272:
6245:
6242:
6237:
6233:
6229:
6224:
6220:
6199:
6194:
6190:
6186:
6183:
6180:
6175:
6171:
6167:
6145:
6141:
6137:
6134:
6114:
6111:
6106:
6102:
6098:
6093:
6089:
6064:
6061:
6056:
6052:
6048:
6043:
6039:
6026:
6023:
6021:
6018:
6003:
5999:
5987:
5986:
5974:
5971:
5967:
5963:
5960:
5957:
5954:
5951:
5948:
5945:
5942:
5939:
5936:
5922:
5921:
5905:
5901:
5897:
5894:
5891:
5888:
5885:
5882:
5877:
5874:
5871:
5868:
5865:
5862:
5859:
5856:
5850:
5847:
5844:
5839:
5836:
5833:
5829:
5825:
5820:
5815:
5811:
5787:
5765:
5762:
5759:
5755:
5751:
5748:
5745:
5742:
5739:
5734:
5730:
5709:
5706:
5703:
5700:
5697:
5694:
5691:
5686:
5682:
5678:
5655:
5652:
5649:
5629:
5609:
5585:
5582:
5579:
5576:
5573:
5570:
5567:
5564:
5561:
5558:
5555:
5552:
5549:
5533:
5530:
5512:
5509:
5506:
5503:
5464:
5444:
5411:
5406:
5402:
5398:
5391:
5387:
5383:
5377:
5374:
5367:
5361:
5356:
5352:
5348:
5341:
5338:
5335:
5331:
5327:
5321:
5318:
5311:
5305:
5300:
5297:
5294:
5290:
5262:
5255:
5252:
5224:
5221:
5212:
5209:
5183:
5176:
5173:
5145:
5142:
5133:
5130:
5110:
5107:
5087:
5082:
5079:
5076:
5072:
5068:
5063:
5059:
5055:
5048:
5045:
5042:
5038:
5034:
5028:
5025:
5018:
5012:
5007:
5004:
5001:
4997:
4993:
4990:
4985:
4981:
4977:
4970:
4967:
4958:
4955:
4929:
4925:
4900:
4897:
4888:
4885:
4859:
4855:
4851:
4846:
4842:
4838:
4832:
4829:
4798:
4795:
4792:
4772:
4767:
4763:
4759:
4756:
4753:
4748:
4744:
4740:
4737:
4732:
4728:
4705:
4701:
4697:
4694:
4672:
4668:
4664:
4658:
4655:
4632:
4629:
4624:
4620:
4616:
4611:
4607:
4586:
4581:
4577:
4573:
4570:
4567:
4562:
4558:
4554:
4551:
4548:
4525:
4522:
4519:
4516:
4512:
4508:
4505:
4502:
4497:
4490:
4487:
4461:
4458:
4455:
4452:
4449:
4446:
4443:
4440:
4435:
4431:
4423:
4419:
4414:
4410:
4407:
4404:
4401:
4396:
4389:
4386:
4358:
4354:
4333:
4318:Jorma Rissanen
4310:plug-in method
4291:
4287:
4264:
4260:
4256:
4251:
4247:
4226:
4221:
4217:
4213:
4210:
4207:
4202:
4198:
4194:
4191:
4186:
4182:
4161:
4158:
4155:
4152:
4147:
4143:
4130:
4127:
4110:
4101:based on data
4090:
4087:
4081:
4078:
4048:
4041:
4038:
4035:
4030:
4027:
4021:
4018:
4010:
4003:
4000:
3997:
3994:
3988:
3984:
3977:
3974:
3971:
3968:
3961:
3957:
3953:
3950:
3946:
3940:
3937:
3934:
3931:
3925:
3922:
3908:
3905:
3892:
3889:
3886:
3883:
3880:
3877:
3874:
3871:
3866:
3862:
3854:
3850:
3845:
3841:
3838:
3835:
3832:
3827:
3824:
3820:
3797:
3793:
3772:
3746:
3743:
3739:
3716:
3712:
3687:
3659:
3656:
3652:
3626:
3623:
3620:
3615:
3612:
3608:
3602:
3599:
3596:
3593:
3587:
3584:
3559:
3555:
3551:
3546:
3542:
3521:
3516:
3512:
3508:
3505:
3502:
3497:
3493:
3489:
3486:
3481:
3477:
3467:be simple and
3456:
3453:
3450:
3447:
3442:
3438:
3425:
3422:
3409:
3406:
3403:
3398:
3394:
3389:
3385:
3382:
3379:
3376:
3373:
3370:
3350:
3347:
3344:
3341:
3336:
3332:
3311:
3306:
3302:
3298:
3295:
3290:
3286:
3273:
3270:
3257:
3254:
3251:
3248:
3245:
3240:
3234:
3211:
3191:
3188:
3185:
3182:
3177:
3173:
3134:
3130:
3090:
3087:
3084:
3072:
3069:
3054:
3051:
3048:
3045:
3042:
3038:
3015:
3012:
3009:
3005:
2984:
2964:
2944:
2940:
2936:
2912:
2886:
2883:
2880:
2876:
2853:
2849:
2826:
2822:
2790:
2787:
2782:
2779:
2776:
2772:
2768:
2763:
2760:
2757:
2753:
2725:
2705:
2701:
2697:
2677:
2668:is bounded by
2657:
2653:
2649:
2625:
2622:
2619:
2616:
2613:
2588:
2585:
2580:
2577:
2574:
2570:
2566:
2561:
2558:
2555:
2551:
2530:
2527:
2522:
2519:
2516:
2512:
2508:
2503:
2500:
2497:
2493:
2468:
2465:
2445:
2442:
2439:
2435:
2414:
2409:
2406:
2403:
2399:
2395:
2392:
2389:
2384:
2380:
2376:
2373:
2368:
2365:
2362:
2358:
2336:
2313:
2310:
2307:
2303:
2298:
2291:
2288:
2258:
2255:
2252:
2248:
2243:
2236:
2233:
2227:
2224:
2220:
2214:
2209:
2206:
2203:
2199:
2195:
2192:
2170:
2148:
2145:
2142:
2138:
2117:
2097:
2094:
2090:
2086:
2083:
2080:
2077:
2074:
2071:
2068:
2064:
2060:
2057:
2054:
2051:
2048:
2028:
2025:
2022:
2017:
2013:
2009:
2006:
2003:
2000:
1997:
1992:
1989:
1986:
1982:
1961:
1941:
1921:
1899:
1895:
1872:
1868:
1847:
1844:
1841:
1838:
1835:
1813:
1809:
1788:
1783:
1779:
1775:
1772:
1769:
1764:
1760:
1756:
1753:
1750:
1726:
1723:
1720:
1717:
1693:
1690:
1687:
1684:
1664:
1661:
1658:
1636:
1631:
1628:
1606:
1602:
1598:
1595:
1575:
1572:
1552:
1532:
1529:
1526:
1523:
1520:
1517:
1494:
1491:
1471:
1468:
1465:
1445:
1425:
1405:
1402:
1399:
1396:
1393:
1390:
1378:
1375:
1360:
1355:
1350:
1346:
1323:
1319:
1298:
1295:
1292:
1287:
1283:
1278:
1274:
1271:
1268:
1265:
1262:
1259:
1237:
1234:
1231:
1228:
1225:
1222:
1219:
1216:
1213:
1208:
1204:
1200:
1197:
1194:
1191:
1188:
1185:
1163:
1143:
1123:
1103:
1100:
1097:
1094:
1091:
1086:
1082:
1078:
1075:
1070:
1033:
1009:
1004:
1000:
996:
993:
988:
984:
963:
938:
935:
932:
927:
923:
917:
914:
911:
908:
902:
899:
877:
857:
837:
832:
828:
824:
821:
816:
812:
799:
796:
783:
759:
756:
753:
749:
745:
723:
719:
709:which rejects
696:
671:
667:
663:
641:
632:
629:
626:
622:
614:
611:
608:
605:
601:
597:
594:
591:
588:
584:
578:
575:
570:
567:
563:
559:
535:
531:
527:
524:
504:
501:
498:
495:
492:
472:
460:
457:
455:
452:
439:is called the
428:
405:
402:
399:
396:
393:
388:
382:
355:
351:
347:
344:
324:
321:
318:
315:
312:
309:
274:
246:
226:
204:
200:
179:
174:
170:
166:
163:
160:
155:
151:
147:
144:
141:
121:
99:
95:
79:
76:
21:Expected value
15:
9:
6:
4:
3:
2:
7875:
7864:
7861:
7859:
7856:
7854:
7851:
7850:
7848:
7834:
7830:
7826:
7822:
7817:
7812:
7809:(6): 062118.
7808:
7804:
7800:
7793:
7785:
7781:
7776:
7771:
7767:
7763:
7758:
7753:
7749:
7745:
7741:
7734:
7732:
7723:
7719:
7715:
7711:
7706:
7701:
7697:
7693:
7689:
7682:
7680:
7678:
7669:
7665:
7661:
7657:
7653:
7649:
7645:
7638:
7636:
7627:
7623:
7619:
7615:
7610:
7605:
7601:
7597:
7593:
7586:
7584:
7575:
7571:
7567:
7563:
7558:
7553:
7549:
7545:
7541:
7534:
7526:
7522:
7518:
7514:
7510:
7506:
7502:
7495:
7487:
7481:
7477:
7473:
7469:
7465:
7458:
7450:
7449:
7441:
7433:
7429:
7425:
7421:
7416:
7411:
7406:
7401:
7397:
7393:
7389:
7382:
7380:
7371:
7367:
7363:
7359:
7355:
7351:
7344:
7336:
7332:
7325:
7323:
7321:
7319:
7317:
7315:
7313:
7304:
7300:
7296:
7288:
7280:
7276:
7272:
7268:
7263:
7258:
7254:
7250:
7246:
7239:
7237:
7228:
7224:
7220:
7216:
7212:
7208:
7204:
7197:
7195:
7186:
7182:
7178:
7174:
7169:
7164:
7160:
7156:
7152:
7145:
7143:
7141:
7139:
7137:
7135:
7133:
7131:
7129:
7127:
7125:
7116:
7112:
7108:
7104:
7099:
7094:
7090:
7086:
7082:
7075:
7071:
7063:
7060:
7056:
7052:
7042:
7028:
7025:
7020:
7016:
7012:
7007:
7003:
6982:
6979:
6976:
6973:
6970:
6961:
6945:
6942:
6939:
6935:
6923:
6920:
6914:
6911:
6889:
6886:
6883:
6879:
6858:
6836:
6832:
6828:
6825:
6820:
6816:
6812:
6807:
6803:
6799:
6794:
6790:
6769:
6766:
6761:
6757:
6753:
6748:
6744:
6734:
6721:
6718:
6710:
6707:
6704:
6700:
6696:
6693:
6690:
6685:
6681:
6670:
6666:
6657:
6645:
6640:
6636:
6632:
6629:
6620:
6604:
6600:
6577:
6573:
6558:
6556:
6551:
6537:
6529:
6525:
6509:
6505:
6501:
6498:
6493:
6489:
6468:
6448:
6440:
6426:
6404:
6401:
6398:
6378:
6369:
6355:
6352:
6344:
6340:
6331:
6319:
6314:
6310:
6306:
6303:
6281:
6277:
6273:
6270:
6263:
6262:stopping time
6259:
6243:
6240:
6235:
6231:
6227:
6222:
6218:
6192:
6188:
6184:
6181:
6178:
6173:
6169:
6143:
6139:
6135:
6132:
6112:
6109:
6104:
6100:
6096:
6091:
6087:
6078:
6062:
6059:
6054:
6050:
6046:
6041:
6037:
6017:
6001:
5997:
5969:
5965:
5961:
5958:
5955:
5946:
5940:
5934:
5927:
5926:
5925:
5903:
5895:
5892:
5889:
5886:
5880:
5875:
5872:
5869:
5866:
5863:
5860:
5857:
5854:
5848:
5845:
5842:
5837:
5834:
5831:
5827:
5823:
5818:
5813:
5809:
5801:
5800:
5799:
5785:
5763:
5760:
5757:
5753:
5749:
5746:
5740:
5732:
5728:
5704:
5701:
5698:
5695:
5692:
5689:
5684:
5680:
5667:
5653:
5650:
5647:
5627:
5607:
5599:
5577:
5574:
5562:
5559:
5556:
5550:
5547:
5539:
5529:
5525:
5510:
5507:
5504:
5501:
5493:
5492:nonparametric
5488:
5486:
5482:
5476:
5462:
5442:
5432:
5430:
5425:
5404:
5400:
5389:
5385:
5381:
5372:
5365:
5354:
5350:
5339:
5336:
5333:
5329:
5325:
5319:
5316:
5309:
5303:
5298:
5295:
5292:
5288:
5276:
5260:
5250:
5222:
5219:
5207:
5181:
5171:
5143:
5140:
5128:
5116:
5106:
5104:
5099:
5080:
5077:
5074:
5070:
5066:
5061:
5057:
5046:
5043:
5040:
5036:
5032:
5026:
5023:
5016:
5010:
5005:
5002:
4999:
4995:
4991:
4983:
4979:
4968:
4965:
4953:
4942:by setting
4927:
4923:
4898:
4895:
4883:
4857:
4853:
4849:
4844:
4840:
4836:
4830:
4827:
4816:
4812:
4796:
4793:
4790:
4765:
4761:
4757:
4754:
4751:
4746:
4742:
4735:
4730:
4726:
4703:
4695:
4692:
4670:
4666:
4662:
4656:
4653:
4630:
4627:
4622:
4618:
4614:
4609:
4605:
4579:
4575:
4571:
4568:
4565:
4560:
4556:
4549:
4546:
4537:
4520:
4514:
4510:
4503:
4495:
4485:
4472:
4456:
4450:
4447:
4441:
4433:
4429:
4421:
4412:
4408:
4402:
4394:
4384:
4372:
4356:
4331:
4323:
4319:
4315:
4311:
4307:
4289:
4285:
4262:
4258:
4254:
4249:
4245:
4219:
4211:
4208:
4205:
4200:
4196:
4189:
4184:
4180:
4156:
4150:
4145:
4141:
4126:
4124:
4108:
4088:
4085:
4076:
4065:
4060:
4046:
4036:
4028:
4025:
4016:
4008:
3998:
3992:
3986:
3982:
3972:
3966:
3959:
3955:
3951:
3948:
3935:
3929:
3923:
3920:
3912:
3904:
3887:
3881:
3878:
3872:
3864:
3860:
3852:
3843:
3839:
3833:
3825:
3822:
3818:
3795:
3770:
3762:
3744:
3741:
3737:
3714:
3710:
3701:
3685:
3677:
3676:
3657:
3654:
3650:
3640:
3621:
3613:
3610:
3606:
3597:
3591:
3585:
3582:
3574:
3557:
3553:
3549:
3544:
3540:
3514:
3506:
3503:
3500:
3495:
3491:
3484:
3479:
3475:
3451:
3445:
3440:
3436:
3421:
3404:
3396:
3392:
3387:
3380:
3374:
3371:
3368:
3345:
3339:
3334:
3330:
3304:
3300:
3293:
3288:
3284:
3269:
3252:
3249:
3246:
3238:
3209:
3186:
3180:
3175:
3171:
3162:
3158:
3154:
3150:
3132:
3128:
3118:
3116:
3115:
3109:
3104:
3088:
3085:
3082:
3068:
3049:
3046:
3043:
3036:
3010:
3003:
2982:
2962:
2942:
2938:
2934:
2926:
2910:
2902:
2881:
2874:
2851:
2847:
2824:
2820:
2810:
2808:
2804:
2788:
2785:
2777:
2770:
2766:
2758:
2751:
2741:
2739:
2723:
2703:
2699:
2695:
2675:
2655:
2651:
2647:
2639:
2623:
2620:
2617:
2614:
2611:
2602:
2586:
2583:
2575:
2568:
2564:
2556:
2549:
2528:
2525:
2517:
2510:
2506:
2498:
2491:
2481:
2479:
2475:
2474:meta-analysis
2464:
2462:
2443:
2440:
2437:
2433:
2407:
2404:
2401:
2397:
2393:
2390:
2387:
2382:
2378:
2371:
2366:
2363:
2360:
2356:
2334:
2311:
2308:
2305:
2301:
2289:
2286:
2274:
2256:
2253:
2250:
2246:
2234:
2231:
2225:
2222:
2218:
2212:
2207:
2204:
2201:
2197:
2193:
2190:
2182:
2168:
2146:
2143:
2140:
2136:
2115:
2092:
2088:
2084:
2081:
2075:
2072:
2069:
2062:
2058:
2055:
2049:
2046:
2023:
2020:
2015:
2011:
2004:
2001:
1998:
1995:
1990:
1987:
1984:
1980:
1959:
1939:
1919:
1897:
1893:
1870:
1866:
1842:
1839:
1836:
1811:
1807:
1781:
1777:
1773:
1770:
1767:
1762:
1758:
1751:
1748:
1740:
1739:nonparametric
1724:
1721:
1718:
1715:
1707:
1691:
1688:
1685:
1682:
1662:
1659:
1656:
1629:
1626:
1604:
1600:
1596:
1593:
1573:
1570:
1550:
1530:
1527:
1524:
1521:
1518:
1515:
1506:
1492:
1489:
1469:
1466:
1463:
1443:
1423:
1400:
1394:
1391:
1388:
1374:
1353:
1348:
1344:
1321:
1317:
1293:
1285:
1281:
1276:
1269:
1263:
1260:
1257:
1248:
1235:
1232:
1226:
1220:
1214:
1206:
1202:
1198:
1195:
1189:
1183:
1175:
1161:
1121:
1095:
1092:
1089:
1084:
1080:
1073:
1058:
1054:
1050:
1045:
1031:
1024:distribution
1023:
1002:
998:
991:
986:
982:
961:
952:
933:
925:
921:
912:
906:
900:
897:
889:
875:
855:
830:
826:
819:
814:
810:
795:
781:
773:
757:
754:
751:
747:
743:
721:
717:
708:
694:
687:
669:
665:
661:
652:
639:
627:
620:
609:
606:
603:
599:
595:
589:
586:
582:
576:
573:
568:
565:
561:
557:
549:
533:
529:
525:
522:
502:
499:
496:
493:
490:
470:
451:
449:
445:
442:
426:
417:
403:
400:
394:
386:
369:
353:
349:
345:
342:
319:
313:
310:
307:
299:
295:
291:
286:
272:
264:
260:
244:
224:
202:
198:
172:
168:
164:
161:
158:
153:
149:
142:
139:
119:
97:
93:
85:
75:
73:
69:
68:Bayes factors
65:
59:
57:
53:
49:
43:
41:
37:
33:
29:
22:
7806:
7802:
7792:
7750:(1): 66–68.
7747:
7743:
7695:
7691:
7651:
7647:
7599:
7595:
7547:
7543:
7533:
7508:
7504:
7494:
7467:
7457:
7447:
7440:
7395:
7391:
7353:
7343:
7334:
7287:
7252:
7248:
7210:
7206:
7158:
7154:
7088:
7084:
7074:
7055:Leonid Levin
7048:
6962:
6735:
6621:
6564:
6561:Construction
6552:
6418:
6370:
6257:
6028:
5988:
5923:
5668:
5537:
5535:
5526:
5491:
5489:
5484:
5481:an R package
5477:
5433:
5426:
5277:
5114:
5112:
5102:
5100:
4538:
4473:
4373:
4314:Philip Dawid
4309:
4305:
4304:): Robbins'
4132:
4122:
4061:
3913:
3910:
3674:
3641:
3575:
3427:
3275:
3160:
3156:
3152:
3119:
3112:
3107:
3102:
3074:
3067:and so on).
3028:to the next
2924:
2900:
2811:
2742:
2737:
2637:
2600:
2482:
2477:
2470:
2275:
2183:
1507:
1380:
1249:
1176:
1056:
1053:Bayes factor
1051:and (b) the
1046:
1021:
953:
890:
801:
772:Type-I error
683:
653:
550:
462:
447:
443:
440:
418:
370:
297:
293:
289:
287:
262:
258:
81:
60:
51:
44:
31:
25:
6481:such that
6260:if for any
6020:E-Processes
3810:such that
3700:convex hull
3157:log-optimal
1586:under all
774:bounded by
654:In words:
298:nonnegative
294:e-statistic
7847:Categories
7705:1912.06116
7557:1610.02351
7405:1912.11436
7262:2010.09686
7168:2210.01948
7098:2009.02824
7066:References
6528:filtration
6417:", or the
6025:Definition
5115:parametric
3075:If we set
2601:definition
2128:for which
2039:, for any
1932:with mean
1706:parametric
888:, and let
290:e-variable
190:with each
132:. Usually
7833:1050-2947
7816:1108.2468
7766:0027-8424
7722:0090-5364
7668:0035-9246
7626:0883-4237
7609:0912.4269
7574:1369-7412
7525:0018-9448
7424:0027-8424
7370:1556-5068
7279:1369-7412
7227:0964-1998
7185:0883-4237
7115:1369-7412
7029:…
6983:…
6943:−
6924:˘
6921:λ
6912:λ
6890:λ
6859:λ
6829:×
6826:⋯
6813:×
6770:…
6719:≤
6708:−
6694:…
6633:∈
6538:α
6510:α
6499:≥
6449:α
6427:α
6353:≤
6345:τ
6307:∈
6282:τ
6271:τ
6258:e-process
6244:…
6182:…
6113:…
6063:…
5893:
5887:−
5873:
5858:−
5846:κ
5835:−
5832:κ
5824:κ
5810:∫
5786:κ
5761:−
5758:κ
5750:κ
5733:κ
5699:κ
5685:κ
5651:≥
5581:∞
5569:→
5508:λ
5463:τ
5382:∣
5376:^
5373:θ
5337:−
5326:∣
5320:˘
5317:θ
5289:∏
5254:¯
5223:˘
5220:θ
5211:¯
5175:¯
5144:˘
5141:θ
5132:¯
5078:−
5067:∣
5044:−
5033:∣
5027:˘
5024:θ
4996:∏
4969:˘
4966:θ
4957:¯
4899:˘
4896:θ
4887:¯
4854:θ
4837:∣
4831:˘
4828:θ
4794:≥
4755:…
4700:Θ
4696:∈
4693:θ
4663:∣
4657:˘
4654:θ
4631:…
4569:…
4489:¯
4457:θ
4434:θ
4418:Θ
4413:∫
4388:¯
4353:Θ
4255:∪
4216:Θ
4212:∈
4209:θ
4201:θ
4133:Now let
4086:∣
4080:^
4077:θ
4026:∣
4020:^
4017:θ
3952:∈
3888:θ
3865:θ
3849:Θ
3844:∫
3823:↶
3792:Θ
3742:↶
3698:unto the
3655:↶
3611:↶
3550:∪
3511:Θ
3507:∈
3504:θ
3496:θ
3250:
2943:α
2789:…
2724:α
2704:α
2676:α
2656:α
2618:α
2587:…
2529:…
2444:λ
2405:−
2391:…
2364:−
2335:λ
2309:−
2290:˘
2287:λ
2254:−
2235:˘
2232:λ
2198:∏
2147:λ
2116:λ
2093:μ
2076:μ
2073:−
2056:−
2050:∈
2047:λ
2024:μ
2021:−
2005:λ
1991:λ
1940:μ
1799:with the
1771:…
1722:λ
1689:λ
1660:≥
1630:∈
1627:λ
1597:∈
1571:≤
1528:λ
1490:≤
1467:−
1250:and then
1236:θ
1227:θ
1207:θ
1199:∫
1142:Θ
1099:Θ
1096:∈
1093:θ
1085:θ
782:α
758:α
755:≤
695:α
640:α
628:∗
621:≤
610:α
607:≤
577:α
569:≥
526:∈
500:≤
497:α
401:≤
346:∈
225:τ
173:τ
162:…
52:e-process
7784:16578652
7432:32631986
7255:: 1–27.
6622:for all
5103:learning
4371:and set
4308:and the
770:, has
515:and all
483:and any
82:Let the
40:p-values
32:e-values
7059:V. Vovk
7045:History
5598:p-value
4597:where
3672:is the
3153:e-power
3103:trivial
2925:outcome
1543:where
1377:As bets
448:e-value
441:e-value
7831:
7782:
7775:335597
7772:
7764:
7720:
7666:
7624:
7572:
7523:
7482:
7430:
7422:
7368:
7295:trials
7277:
7225:
7183:
7113:
6419:level-
6079:. Let
3642:where
2276:where
1885:, the
637:
616:
259:sample
7811:arXiv
7700:arXiv
7698:(3).
7604:arXiv
7602:(1).
7552:arXiv
7400:arXiv
7257:arXiv
7163:arXiv
7161:(4).
7093:arXiv
6782:with
5720:with
5239:or
3428:Let
1619:and
296:is a
7829:ISSN
7780:PMID
7762:ISSN
7718:ISSN
7664:ISSN
7622:ISSN
7570:ISSN
7521:ISSN
7480:ISBN
7428:PMID
7420:ISSN
7366:ISSN
7275:ISSN
7223:ISSN
7181:ISSN
7111:ISSN
6592:for
5702:<
5696:<
4915:for
4783:for
3783:on
3702:of
3678:of
3322:and
3276:Let
2805:and
2638:ever
2621:<
2615:<
1336:vs.
1022:some
802:Let
494:<
7821:doi
7770:PMC
7752:doi
7710:doi
7656:doi
7614:doi
7562:doi
7513:doi
7472:doi
7410:doi
7396:117
7358:doi
7299:doi
7267:doi
7215:doi
7211:184
7173:doi
7103:doi
6405:100
6256:an
5950:min
5890:log
5870:log
5640:if
5490:In
5160:or
5113:In
4344:on
4322:MDL
4066:)
3945:sup
3247:log
3159:or
1057:are
736:if
292:or
288:An
261:or
26:In
7849::
7827:.
7819:.
7807:84
7805:.
7801:.
7778:.
7768:.
7760:.
7748:58
7746:.
7742:.
7730:^
7716:.
7708:.
7696:49
7694:.
7690:.
7676:^
7662:.
7652:55
7650:.
7646:.
7634:^
7620:.
7612:.
7600:26
7598:.
7594:.
7582:^
7568:.
7560:.
7548:80
7546:.
7542:.
7519:.
7509:30
7507:.
7503:.
7478:.
7470:.
7466:.
7426:.
7418:.
7408:.
7394:.
7390:.
7378:^
7364:.
7356:.
7352:.
7333:.
7311:^
7273:.
7265:.
7253:86
7251:.
7247:.
7235:^
7221:.
7209:.
7205:.
7193:^
7179:.
7171:.
7159:38
7157:.
7153:.
7123:^
7109:.
7101:.
7089:84
7087:.
7083:.
6368:.
6016:.
5947::=
5798::
5747::=
5431:.
5098:.
4850::=
4536:.
4409::=
3903:.
3586::=
3108:if
3086::=
2809:.
2740:.
2463:.
2273:,
2194::=
1996::=
1519::=
1354::=
901::=
416:.
58:.
30:,
7835:.
7823::
7813::
7786:.
7754::
7724:.
7712::
7702::
7670:.
7658::
7628:.
7616::
7606::
7576:.
7564::
7554::
7527:.
7515::
7488:.
7474::
7434:.
7412::
7402::
7372:.
7360::
7337:.
7305:.
7301::
7281:.
7269::
7259::
7229:.
7217::
7187:.
7175::
7165::
7117:.
7105::
7095::
7026:,
7021:2
7017:E
7013:,
7008:1
7004:E
6980:,
6977:2
6974:,
6971:1
6946:1
6940:i
6936:X
6931:|
6915:=
6887:,
6884:i
6880:E
6837:n
6833:E
6821:2
6817:E
6808:1
6804:E
6800:=
6795:n
6791:M
6767:,
6762:2
6758:M
6754:,
6749:1
6745:M
6722:1
6716:]
6711:1
6705:i
6701:X
6697:,
6691:,
6686:1
6682:X
6677:|
6671:i
6667:E
6663:[
6658:P
6652:E
6646::
6641:0
6637:H
6630:P
6605:i
6601:X
6578:i
6574:S
6506:/
6502:1
6494:n
6490:E
6469:n
6402:=
6399:n
6379:n
6356:1
6350:]
6341:E
6337:[
6332:P
6326:E
6320::
6315:0
6311:H
6304:P
6278:E
6274:,
6241:,
6236:2
6232:E
6228:,
6223:1
6219:E
6198:)
6193:n
6189:X
6185:,
6179:,
6174:1
6170:X
6166:(
6144:n
6140:E
6136:,
6133:n
6110:,
6105:2
6101:E
6097:,
6092:1
6088:E
6060:,
6055:2
6051:X
6047:,
6042:1
6038:X
6002:1
5998:H
5985:.
5973:)
5970:t
5966:/
5962:1
5959:,
5956:1
5953:(
5944:)
5941:t
5938:(
5935:f
5904:2
5900:)
5896:p
5884:(
5881:p
5876:p
5867:p
5864:+
5861:p
5855:1
5849:=
5843:d
5838:1
5828:p
5819:1
5814:0
5764:1
5754:p
5744:)
5741:p
5738:(
5729:f
5708:}
5705:1
5693:0
5690::
5681:f
5677:{
5654:g
5648:f
5628:g
5608:f
5584:]
5578:,
5575:0
5572:[
5566:]
5563:1
5560:,
5557:0
5554:[
5551::
5548:f
5511:U
5505:+
5502:1
5443:n
5410:)
5405:n
5401:X
5397:(
5390:n
5386:X
5366:q
5360:)
5355:i
5351:X
5347:(
5340:1
5334:i
5330:X
5310:q
5304:n
5299:1
5296:=
5293:i
5261:W
5251:q
5208:q
5182:W
5172:q
5129:q
5086:)
5081:1
5075:i
5071:X
5062:i
5058:X
5054:(
5047:1
5041:i
5037:X
5017:q
5011:n
5006:1
5003:=
5000:i
4992:=
4989:)
4984:n
4980:X
4976:(
4954:q
4928:n
4924:X
4884:q
4858:0
4845:0
4841:X
4797:0
4791:i
4771:)
4766:i
4762:X
4758:,
4752:,
4747:1
4743:X
4739:(
4736:=
4731:i
4727:X
4704:1
4671:i
4667:X
4628:,
4623:2
4619:X
4615:,
4610:1
4606:X
4585:)
4580:n
4576:X
4572:,
4566:,
4561:1
4557:X
4553:(
4550:=
4547:Y
4524:)
4521:Y
4518:(
4515:p
4511:/
4507:)
4504:Y
4501:(
4496:W
4486:q
4460:)
4454:(
4451:W
4448:d
4445:)
4442:Y
4439:(
4430:q
4422:1
4406:)
4403:Y
4400:(
4395:W
4385:q
4357:1
4332:W
4290:1
4286:H
4263:1
4259:H
4250:0
4246:H
4225:}
4220:1
4206::
4197:Q
4193:{
4190:=
4185:1
4181:H
4160:}
4157:P
4154:{
4151:=
4146:0
4142:H
4109:Y
4089:Y
4047:)
4040:)
4037:Y
4034:(
4029:Y
4009:p
4002:)
3999:Y
3996:(
3993:q
3987:=
3983:(
3976:)
3973:Y
3970:(
3967:p
3960:0
3956:H
3949:P
3939:)
3936:Y
3933:(
3930:q
3924:=
3921:E
3891:)
3885:(
3882:W
3879:d
3876:)
3873:Y
3870:(
3861:p
3853:0
3840:=
3837:)
3834:Y
3831:(
3826:Q
3819:p
3796:0
3771:W
3745:Q
3738:p
3715:0
3711:H
3686:Q
3658:Q
3651:p
3625:)
3622:Y
3619:(
3614:Q
3607:p
3601:)
3598:Y
3595:(
3592:q
3583:E
3558:1
3554:H
3545:0
3541:H
3520:}
3515:0
3501::
3492:P
3488:{
3485:=
3480:0
3476:H
3455:}
3452:Q
3449:{
3446:=
3441:1
3437:H
3408:)
3405:Y
3402:(
3397:0
3393:p
3388:/
3384:)
3381:Y
3378:(
3375:q
3372:=
3369:E
3349:}
3346:Q
3343:{
3340:=
3335:1
3331:H
3310:}
3305:0
3301:P
3297:{
3294:=
3289:0
3285:H
3256:]
3253:E
3244:[
3239:Q
3233:E
3210:S
3190:}
3187:Q
3184:{
3181:=
3176:1
3172:H
3133:1
3129:H
3089:1
3083:E
3053:)
3050:1
3047:+
3044:j
3041:(
3037:Y
3014:)
3011:j
3008:(
3004:Y
2983:Y
2963:Y
2939:/
2935:1
2911:Y
2885:)
2882:j
2879:(
2875:Y
2852:i
2848:X
2825:i
2821:X
2786:,
2781:)
2778:2
2775:(
2771:Y
2767:,
2762:)
2759:1
2756:(
2752:Y
2700:/
2696:1
2652:/
2648:1
2624:1
2612:0
2584:,
2579:)
2576:2
2573:(
2569:Y
2565:,
2560:)
2557:1
2554:(
2550:Y
2526:,
2521:)
2518:2
2515:(
2511:E
2507:,
2502:)
2499:1
2496:(
2492:E
2441:,
2438:i
2434:E
2413:)
2408:1
2402:i
2398:X
2394:,
2388:,
2383:1
2379:X
2375:(
2372:=
2367:1
2361:i
2357:X
2312:1
2306:i
2302:X
2297:|
2257:1
2251:i
2247:X
2242:|
2226:,
2223:i
2219:E
2213:n
2208:1
2205:=
2202:i
2191:E
2169:Y
2144:,
2141:i
2137:E
2096:]
2089:/
2085:1
2082:,
2079:)
2070:1
2067:(
2063:/
2059:1
2053:[
2027:)
2016:i
2012:X
2008:(
2002:+
1999:1
1988:,
1985:i
1981:E
1960:P
1920:P
1898:i
1894:X
1871:0
1867:H
1846:]
1843:1
1840:,
1837:0
1834:[
1812:i
1808:X
1787:)
1782:n
1778:X
1774:,
1768:,
1763:1
1759:X
1755:(
1752:=
1749:Y
1725:U
1719:+
1716:1
1692:U
1686:+
1683:1
1663:0
1657:E
1635:R
1605:0
1601:H
1594:P
1574:0
1551:U
1531:U
1525:+
1522:1
1516:E
1493:0
1470:1
1464:E
1444:E
1424:E
1404:)
1401:Y
1398:(
1395:E
1392:=
1389:E
1359:Q
1349:1
1345:H
1322:0
1318:H
1297:)
1294:Y
1291:(
1286:0
1282:p
1277:/
1273:)
1270:Y
1267:(
1264:q
1261:=
1258:E
1233:d
1230:)
1224:(
1221:w
1218:)
1215:Y
1212:(
1203:q
1196:=
1193:)
1190:Y
1187:(
1184:q
1162:Q
1122:w
1102:}
1090::
1081:Q
1077:{
1074:=
1069:Q
1032:Q
1008:}
1003:0
999:P
995:{
992:=
987:0
983:H
962:E
937:)
934:Y
931:(
926:0
922:p
916:)
913:Y
910:(
907:q
898:E
876:Y
856:Q
836:}
831:0
827:P
823:{
820:=
815:0
811:H
752:E
748:/
744:1
722:0
718:P
707:,
670:E
666:/
662:1
631:)
625:(
613:)
604:E
600:/
596:1
593:(
590:P
587:=
583:)
574:1
566:E
562:(
558:P
534:0
530:H
523:P
503:1
491:0
471:E
444:.
427:E
404:1
398:]
395:E
392:[
387:P
381:E
354:0
350:H
343:P
323:)
320:Y
317:(
314:E
311:=
308:E
273:Y
245:Y
203:i
199:X
178:)
169:X
165:,
159:,
154:1
150:X
146:(
143:=
140:Y
120:Y
98:0
94:H
23:.
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.