/
Текст
Asymptotic Statistics
This book is an introduction to the field of asymptotic statistics. The
treatment is both practical and mathematically rigorous. In addition to
most of the standard topics of an asymptotics course, including like-
lihood inference, M-estimation, asymptotic efficiency, U-statistics, and
rank procedures, the book also presents recent research topics such as
semiparametric models, the bootstrap, and empirical processes and their
applications.
One of the unifying themes is the approximation by limit experi-
ments. This entails mainly the local approximation of the classical i.i.d.
set-up with smooth parameters by location experiments involving a sin-
gle, normally distributed observation. Thus, even the standard subjects
of asymptotic statistics are presented in a novel way.
Suitable as a text for a gradu ate or Master' s level statistics course, this
book also gives researchers in statistics, probability, and their applications
an overview of the latest research in asymptotic statistics.
A.W. van der Vaart is Professor of Statistics in the Department of
Mathematics and Computer Science at the Vrije Universiteit, Amsterdam.
CAMBRIDGE SERIES IN STATISTICAL AND PROBABILISTIC MATHEMATICS
Editorial Board:
R. Gill, Department of Mathematics, Utrecht University
B.D. Ripley, Department of Statisties, University of Oxford
S. Ross, Department of lndustrial Engineering, University of California, Berkeley
M. Stein, Department of Statisties, University of Chicago
D. Williams, School of Mathematical Sciences, University of Bath
This series of high-quality upper-division textbooks and expository monographs covers
all aspects of stochastic applicable mathematics. The topics range from pure and applied
statistics to probability theory, operations research, optimization, and mathematical pro-
gramming. The books contain clear presentations of new developments in the field and
also of the state of the art in classical methods. While emphasizing rigorous treatment of
theoretical methods, the books also contain applications and discussions of new techniques
made possible by advances in computational practice.
Already published
1. Bootstrap Methods and Their Application, by A.C. Davison and D.V. Hinkley
2. Markov Chains, by J. Norris
Asymptotic Statistics
A.W. VAN DER VAART
"", "", CAMBRIDGE
; UNIVERSITY PRESS
PUBLISHED BY THE PRESS SYNDICATE OF THE UNIVERSITY OF CAMBRIDGE
The Pitt Building, Trumpington Street, Cambridge, United Kingdom
CAMBRIDGE UNIVERSITY PRESS
The Edinburgh Building, Cambridge CB2 2RU, UK http://www.eup.eam.ae.uk
40 West 20th Street, New York, NY 10011-4211, USA http://www.cup.org
10 Stamford Road, Oakleigh, Melbourne 3166, Australia
Ruiz de Alarc6n 13, 28014 Madrid, Spain
<9 Cambridge University Press 1998
This book is in copyright. Subject to statutory exeeption and
to the provisions of relevant collective lieensing agreements,
no reproduction of any part may take place without
the written permission of Cambridge University Press.
First published 1998
First paperbaek edition 2000
Printed in the United States of America
Typeset in Times Roman 10/12.5 pt in Lkfff(2 [TB]
A catalog record for this book is available from the British Library
Library of Congress Cataloging in Publication data
Vaart, A. W. van der
Asymtotic statistics / A.W. van der Vaart.
p. cm. - (Cambridge series in statistical and probablistic
mathematies)
Includes bibliographical referenees.
1. Mathematical statistics - Asymptotic the ory. I. Title.
II. Series: eambridge series on statistical and probablistic
mathematies.
CA276.V22 1998
519.5-dc21 98-15176
ISBN O 521 49603 9 hardback
ISBN O 521 78450 6 paperbaek
To Maryse and Marianne
Contents
Preface page XUl
Notation page xv
1. Introduction 1
1.1. Approximate Statistical Procedures 1
1.2. Asymptotic Optimality Theory 2
1.3. Limitations 3
1.4. The Index n 4
2. Stochastic Convergence 5
2.1. Basic Theory 5
2.2. Stochastic o and O Symbols 12
*2.3. Characteristic Functions 13
*2.4. Almost-Sure Representations 17
*2.5. Convergence of Moments 17
*2.6. Convergence- Determining Classes 18
*2.7. Law of the Iterated Logarithm 19
*2.8. Lindeberg-Feller Theorem 20
*2.9. Convergence in Total Variation 22
Problems 24
3. Delta Method 25
3.1. Basic Result 25
3.2. Variance-S tabilizing Transformations 30
*3.3. Higher-Order Expansions 31
*3.4. Uniform Delta Method 32
*3.5. Moments 33
Problems 34
4. Moment Estimators 35
4.1. Method of Moments 35
*4.2. Exponential Families 37
Problem s 40
5. M- and Z-Estimators 41
5.1. lntroduction 41
5.2. Consistency 44
5.3. Asymptotic Normality 51
Vll
VIH Contents
* 5.4. Estimated Parameters 60
5.5. Maximum Likelihood Estimators 61
*5.6. Classical Conditions 67
*5.7. One-Step Estimators 71
*5.8. Rates of Convergence 75
*5.9. Argmax Theorem 79
Problems 83
6. Contiguity 85
6.1. Likelihood Ratios 85
6.2. Contiguity 87
Problems 91
7. Local Asymptotic Normality 92
7.1. Introduction 92
7.2. Expanding the Likelihood 93
7.3. Convergence to a Normal Experiment 97
7.4. Maximum Likelihood 100
*7.5. Limit Distributions under Alternatives 103
*7.6. Local Asymptotic Normality 103
Problems 106
8. Efficiency of Estimators 108
8.1. Asymptotic Concentration 108
8.2. Relative Efficiency 110
8.3. Lower Bound for Experiments 111
8.4. Estimating Normal Means 112
8.5. Convolution Theorem 115
8.6. Almost - Everywhere Convolution
Theorem 115
*8.7. Local Asymptotic Minimax Theorem 117
*8.8. Shrinkage Estimators 119
*8.9. Achieving the Bound 120
*8.10. Large Deviations 122
Problems 123
9. Limits of Experiments 125
9.1. Introduction 125
9.2. Asymptotic Representation Theorem 126
9.3. Asymptotic Normality 127
9.4. Uniform Distribution 129
9.5. Pareto Distribution 130
9.6. Asymptotic Mixed Normality 131
9.7. Heuristics 136
Problems 137
10. Bayes Procedures 138
10.1. Introduction 138
10.2. Bernstein-von Mises Theorem 140
C ontents IX
10.3. Point Estimators 146
* 10.4. Consistency 149
Problems 152
11. Projections 153
11.1. Projections 153
11.2. Conditional Expectation 155
11.3. Projection onto Sums 157
* 11.4. Hoeffding Decomposition 157
Problem s 160
12. U -Statistics 161
12.1. One-Sample U -Statistics 161
12.2. Two-Sample U -statistics 165
* 12.3. Degenerate U -Statistics 167
Problem s 171
13. Rank, Sign, and Permutation Statistics 173
13.1. Rank Statistics 173
13.2. Signed Rank Statistics 181
13.3. Rank Statistics for lndependence 184
* 13.4. Rank Statistics under Altematives 184
13.5. Permutation Tests 188
*13.6. Rank Central Limit Theorem 190
Problems 190
14. Relative Efficiency of Tests 192
14.1. Asymptotic Power Functions 192
14.2. Consistency 199
14.3. Asymptotic Relative Efficiency 201
* 14.4. Other Relative Efficiencies 202
*14.5. Rescaling Rates 211
Problems 213
15. Efficiency of Tests 215
15.1. Asymptotic Representation Theorem 215
15.2. Testing Normal Means 216
15.3. Local Asymptotic Normality 218
15.4. One-Sample Location 220
15.5. Two-Sample Problems 223
Problems 226
16. Likelihood Ratio Tests 227
16.1. lntroduction 227
* 16.2. Taylor Expansion 229
16.3. Using Local Asymptotic Normality 231
16.4. Asymptotic Power Functions 236
x Contents
16.5. B artlett Correction 238
* 16.6. Bahadur Efficiency 238
Problem s 241
17. Chi -Square Tests 242
17.1. Quadratic Forms in N ormal Vectors 242
17.2. Pearson Statistic 242
17.3. Estimated Parameters 244
17.4. Testing Independence 247
*17.5. Goodness-of-Fit Tests 248
* 17.6. Asymptotic Efficiency 251
Problems 253
18. Stochastic Convergence in Metric Spaces 255
18.1. Metric and Normed Spaces 255
18.2. Basic Properties 258
18.3. Bounded Stochastic Processes 260
Problems 263
19. Empirical Processes 265
19.1. Empirical Distribution Functions 265
19.2. Empirical Distributions 269
19.3. Goodness-of-Fit Statistics 277
19.4. Random Functions 279
19.5. Changing Classes 282
19.6. Maximallnequalities 284
Problems 289
20. Functional Delta Method 291
20.1. von Mises Calculus 291
20.2. Hadamard- Differentiable Functions 296
20.3. Some Examples 298
Problem s 303
21. Quantiles and Order Statistics 304
21.1. Weak Consistency 304
21.2. Asymptotic Normality 305
21.3. Median Absolute Deviation 310
21.4. Extreme Values 312
Problems 315
22. L-Statistics 316
22.1. Introduction 316
22.2. Hajek Projection 318
22.3. Delta Method 320
22.4. L - Estimators for Location 323
Problem s 324
23. Bootstrap 326
Contents XI
23.1. lntroduction 326
23.2. Consistency 329
23.3. Higher-Order Correctness 334
Problems 339
24. Nonparametric Density Estimation 341
24.1 lntroduction 341
24.2 Kemel Estimators 341
24.3 Rate Optimality 346
24.4 Estimating a Unimodal Density 349
Problems 356
25. Semiparametric Models 358
25.1 lntroduction 358
25.2 Banach and Hilbert Spaces 360
25.3 Tangent Spaces and Information 362
25.4 Efficient Score Functions 368
25.5 Score and lnformation Operators 371
25.6 Testing 384
*25.7 Efficiency and the Delta Method 386
25.8 Efficient Score Equations 391
25.9 General Estimating Equations 400
25.10 Maximum Likelihood Estimators 402
25.11 Approximately Least-Favorable
Submodels 408
25.12 Likelihood Equations 419
Problems 431
References 433
Index 439
Preface
This book grew out of courses that I gave at various places, including a graduate course in
the Statistics Department of Texas A&M University, Master' s level courses for mathematics
students specializing in statistics at the Vrije Universiteit Amsterdam, acourse in the DEA
program (graduate level) of Universite de Paris-sud, and courses in the Dutch AIO-netwerk
(graduate level) .
The mathematicallevel is mixed. Some parts I have used for second year courses for
mathematics students (but they find it tough), other parts I would only recommend for a
gradu ate program. The text is written both for students who know about the technical
details of measure theory and probability, but little about statistics, and vice versa. This
requires brief explanations of statistical methodology, for instance of what a rank test or
the bootstrap is about, and there are similar excursions to introduce mathematical details.
Familiarity with (higher-dimensional) calculus is necessary in all of the manuscript. Metric
and normed spaces are briefly introduced in Chapter 18, when these concepts become
necessary for Chapters 19, 20, 21 and 22, but I do not expect that this would be enough as a
first introduction. For Chapter 25 basic knowledge of Hilbert spaces is extremely helpful,
although the bare essentials are summarized at the beginning. Measure theory is implicitly
assumed in the whole manuscript but can at most places be avoided by skipping proofs, by
ignoring the word "measurable" or with a bit of handwaving. Because we deal mostly with
i.i.d. observations, the simplest limit theorems from probability theory suffice. These are
derived in Chapter 2, but prior exposure is helpful.
Sections, resuIts or proofs that are preceded by asterisks are either of secondary impor-
tance ar are out of line with the natural order of the chapters. As the chart in Figure 0.1
shows, many of the chapters are independent from one another, and the book can be used
for several different courses.
A unifying theme is approximation by alimit experiment. The full theory is not developed
(another writing project is on its way), but the material is limited to the "weak topology"
on experiments, which in 90% of the book is exemplified by the case of smooth parameters
of the distribution of i.i.d. observations. For this situation the theory can be developed
by relatively simple, direct arguments. Limit experiments are used to explain efficiency
properties, but also why certain procedure s asymptotically take a certain form.
A second major theme is the application of results on abstract empirical processes. These
already have benefits for deriving the usual theorems on M -estimators for Euclidean pa-
rameters but are indispensable if discussing more involved situations, such as M -estimators
with nuisance parameters, chi -square statistics with data-dependent cells, or semiparamet-
ric models. The general theory is summarized in about 30 pages, and it is the applications
Xl11
XIV
Preface
Figure 0.1. Dependence chart. A solid arrow means that a chapter is a prerequisite for a next chapter.
A dotted arrow means a natural continuation. Vertical or horizontal position has no independent
meanlng.
that we focus on. In a sense, it would have been better to place this material (Chapters
18 and 19) earlier in the book, but instead we start with material of more direct statistical
relevance and of a less abstract character. A drawback is that a few (starred) proofs point
ahead to later chapters.
Almost every chapter end s with a "Notes" section. These are meant to give a rough
historical sketch, and to provide entries in the literature for further reading. They certainly
do not give sufficient credit to the original contributions by many authors and are not meant
to serve as references in this way.
Mathematical statistics obtains its relevanee from applications. The subjects of this book
have been chosen accordingly. On the other hand, this is a mathematician's book in that
we have made some effort to present results in a nice way, without the (unnecessary) lists
of "regularity conditions" that are sometimes found in statistics books. Occasionally, this
means that the accompanying proof must be more involved. If this means that an idea could
go lost, then an informal argument precedes the statement of a result.
This does not mean that I have strived after the greatest possible generality. A simple,
clean presentation was the main aime
Leiden, September 1997
A.W. van der Vaart
N otation
A*
Iff>*
Cb(T), UC(T), C(T)
(,OO(T)
Lr (Q), Lr (Q)
IlfllQ,r
IIzlloo,llzllT
lin
C,N,Q,IR,Z
EX, E*X, var X, sdX, Cov X
JID n , CG n
G p
N (jL, b), t n , X;
2
Za, X n a' tn,a
,
«
<:J,<:J [>
<
('...J
'v'-t
p
as
N (8, T, d), N[] (8, T, d)
J (8, T, d), J[] (8, T, d)
op(l),Op(l)
adjoint operator
dual space
(bounded, uniformly) continuous functions on T
bounded functions on T
measurable functions whose rth powers are Q-integrable
norm of Lr(Q)
uniform norm
linear span
number fields and sets
( out er) expectation, variance, standard deviation,
covariance (matrix) of X
empirical measure and process
p - Brownian bridge
normal, t and chisquare distribution
upper a-quantiles of normal, chisquare and t distributions
absolutely continuous
contiguous, mutually contiguous
smaller than up to a constant
convergence in distribution
convergence in probabi1ity
convergence almost surely
covering and bracketing number
entropy integral
stochastic order symbols
xv
1
Introduction
Why asymptotic statistics? The use of asymptotic approximations is two-
fold. First, they enable us to find approximate tests and confidence regions.
Second, approximations can be us ed theoretically to study the quality
( efficiency) of statistical procedures.
1.1 Approximate Statistical Procedures
To carry out a statistical test, we need to know the critical value for the test statistic. In
most cases this means that we must know the distribution of the test statistic under the
null hypothesis. Sometimes this is known exactly, but more often only approximations are
available. This may be because the distribution of the statistic is analytically intractable,
or perhaps the postulated statistical model is considered only an approximation of the true
underlying distributions. In both cases the use of an approximate critical value may be fully
satisfactory for practical purposes.
Consider for instance the classical t-test for location. Given a sample of independent
observations XI, . . . , X n, we wish to test a null hypothesis conceming the mean /-L == EX.
The t-test is based on the quotient of the sample mean X n and the sample standard deviation
Sn. If the observations arise from a normal distribution with mean /-Lo, then the distribution
of ,Jii ( X n - /-Lo) / Sn is known exactly: It is a t -distribution with n-I degrees of freedom.
However, we may have doubts regarding the normality, or we might even believe in a
completely different model. If the number of observations is not too small, this does not
matter too much. Then we may act as if ,Jii( X n - /-Lo)/ Sn possesses a standard normal
distribution. The theoretical justification is the limiting result, as n -+ 00,
( ,Jii( Xn - /-L) )
sUpPJL < x -<P(x) -+0,
x Sn
provided the variables Xi have a finite second moment. This variation on the central limit
theorem is proved in the next chapter. A "large sample" level ex test is to reject Ha : /-L == /-La
if 1,Jii ( X n - /-Lo) / Sn I exceeds the upper ex /2 quantile of the standard normal distribution.
Table 1.1 gives the significance level of this test if the observations are either normally or
exponentially distributed, and ex == 0.05. For n > 20 the approximation is quite reasonable
in the normal case. If the underlying distribution is exponential, then the approximation is
less satisfactory, because of the skewness of the exponential distribution.
1
2
Introduction
Table 1.1. Level of the test with critical region
1,Jn( X n - fLO)/ Sn I > 1.96 ifthe observations
are sampledfrom the normal ar
exponential distribution.
n Normal Exponential a
5 0.122 0.19
10 0.082 0.14
15 0.070 0.11
20 0.065 0.10
25 0.062 0.09
50 0.056 0.07
100 0.053 0.06
a The third eolumn gives approximations based on 10,000
simulations.
In many ways the t -test is an uninteresting example. There are many other reasonable
test statistics for the same problem. Often their nulI distributions are difficult to ealculate.
An asymptotic result similar to the one for the t -statistic would make them practicalIy
applicable at least for large sample sizes. Thus, one aim of asymptotic statistics is to derive
the asymptotic distribution of many types of statistics.
There are similar benefits when obtaining confidence intervals. For instance, the given
approximation result asserts that ,Jn ( X n - J1-) / Sn is approximately standard normalI y dis-
tributed if J1- is the true mean, whatever its value. This means that, with probability approx-
imately 1 - 2a,
,Jn( X n - J1-)
- z < < Z
a - Sn - a.
This can be rewritten as the confidence statement J1- == X n ::!:: Za Sn / ,Jn in the usual manner.
For large n its confidence level should be close to 1 - 2a.
As another example, consider maximum likelihood estimators en based on a sample of
size n from a density Pe. A major result in asymptotic statisties is that in many situations
,Jn (en - e) is asymptoticalIy normalIy distributed with zero mean and eovariance matrix the
inverse of the Fisher information matrix le. If Z is k-variate normalIy distributed with mean
zero and nonsingular covariance matrix 'b, then the quadratic form ZT -l Z possesses a
ehi-square distribution with k degrees of freedom. Thus, acting as if ,Jn(e n - e) possesses
an Nk (O, le-I) distribution, we find that the elIipsoid
{ e : ce - enl len ce - en) < X a }
is an approximate 1 - a confidence region, if Xf,a is the appropriate critical value from the
chi -square distribution. A closely related alternative is the regi on based on inverting the
likelihood ratio test, which is also based on an asymptotic approximation.
1.2 Asymptotic Optimality Theory
For a relatively smalI number of statistical problems there exists an exact, optimal solution.
For instance, the Neyman-Pearson theory leads to optimal (uniformIy most powerful) tests
1.3 Limitations
3
in certain exponential family models; the Rao-Blackwell theory allows us to conclude that
certain estimators are of minimum variance among the unbiased estimators. An important
and fairly general result is the Cramer-Rao bound for the variance of unbiased estimators,
but it is often not sharp.
If exact optimality theory does not give resu1ts, be it because the problem is untractable
or because there exist no "optimal" procedures, then asymptotic optimality the ory may
help. For instance, to compare two tests we might compare approximations to their power
functions. To compare estimators, we might compare asymptotic variances rather than
exact variances. A major resu1t in this area is that for smooth parametric model s maximum
likelihood estimators are asymptotically optimal. This roughly means the following. First,
maximum likelihood estimators are asymptotically consistent: The sequence of estimators
converges in probability to the true value of the parameter. Second, the rate at which
maximum likelihood estimators converge to the true value is the fastest possible, typically
1/ ,Jii. Third, their asymptotic variance, the variance of the limit distribution of ,Jii (en - e),
is minimal; in fact, maximum likelihood estimators "asymptotically attain" the Cramer-Rao
bound. Thus asymptotics justify the use of the maximum likelihood method in certain
situations. It is of interest here that, even though the method of maximum likelihood often
leads to reasonable estimators and has great intuitive appeal, in general it does not lead
to best estimators for finite samples. Thus the use of an asymptotic criterion simplifies
optimality theory considerably.
By taking limits we can gain much insight in the structure of statistical experiments. It
turns out that not only estimators and test statistics are asymptoticalIy normalIy distributed,
but often also the whole sequence of statistical models converges to a model with a nor-
mal observation. Our good understanding of the latter "canonical experiment" translates
directly into understanding other experiments asymptotically. The mathematical beauty of
this theory is an added benefit of asymptotic statistics. Though we shalI be mostly concemed
with nonnallimiting theory, this theory applies equalIy well to other situations.
1.3 Limitations
Although asymptotics is both practicalIy useful and of theoretical importance, it should not
be taken for more than what it is: approximations. Clearly, a theorem that can be interpreted
as saying that a statistical procedure works fine for n ex) is of no use if the number of
available observations is n == 5.
In fact, strictly speaking, most asymptotic resu1ts that are currently available are logically
useless. This is because most asymptotic resuIts are limit resu1ts, rather than approximations
consisting of an approximating formula plus an accurate error bound. For instance, to
estimate a value a, we consider it to be the 25th element a == a25 in a sequence al, a2, . . . ,
and next take limn oo an as an approximation. The accuracy of this procedure depends
crucially on the choice of the sequence in which a25 is embedded, and it seems impossible
to defend the procedure from a logical point of view. This is why there is good asymptotics
and bad asymptotics and why two types of asymptotics sometimes lead to confiicting
claims.
Fortunately, many limit resuits of statistics do give reasonable answers. Because it may
be theoretically very hard to ascertain that approximation errors are small, one often takes
recourse to simulatian studies to judge the accuracy of a certain approximation.
4
Introduction
Just as care is neededifusing asymptotic results for approximations, results on asyrnptotic
optimality must be judged in the right manner. One pitfall is that even though a certain
procedure, such as maximum likelihood, is asymptotically optimal, there may be many
other procedures that are asymptotically optimal as well. For finite samples these may
behave differently and possibly better. Then so-called higher-order asymptotics, which
yield better approximations, may be fruitful. See e.g., [7], [52] and [114]. Although we
occasionally touch on this subject, we shalI mostly be concerned with what is known as
"first -order asymptotics."
1.4 The Index n
In all of the following n is an index that tends to infinity, and asymptotics means taking
limits as n 00. In most situations n is the number of observations, so that usually
asymptotics is equivalent to "large-sample theory." However, certain abstract results are
pure limit theorems that have nothing to do with individual observations. In that case n just
plays the role of the index that goes to infinity.
1.5 Notation
A symbol index is given on page xy.
For brevity we often use operator notation for evaluation of expectations and have special
symbols for the empirical measure and process.
For P a measure on a measurable space (X, B) and I : X r--+ JRk a measurable function,
p I denotes the integral J I d P; equivalently, the expectation Ep f(X 1) for XI arandom
variable distributed according to P. When applied to the empirical measure ltD n of a sample
XI, . . . , X n , the discrete uniform meas ure on the sample values, this yields
1 n
ltDnl == - LI(X i ).
n. 1
1=
This formula can also be viewed as simply an abbreviation for the average on the right. The
empirical process G n I is the centered and scaled version of the empirical measure, defined
by
1 n
Gnl == y'n(JPnl - PI) == L(I(Xi) - Epf(X i )).
n. 1
1=
This is studied in detail in Chapter 19, but is used as an abbreviation throughout the book.
2
Stochastic Convergence
This chapter provides a review ofbasic modes of convergence of sequences
of stochastic vectors, in particular convergence in distribution and in
probability.
2.1 Basic Theory
Arandom vector in JRk is a vector X == (XI,..., X k ) of real random variables. t The dis-
tributionfunction of X is the map x P(X < x).
A sequence of random vectors X n is said to converge in distribution to arandom vector
X if
P(X n < x) --+ P(X < x),
for every x at which the limit distribution function x P(X < x) is continuous. Alterna-
tive names are weak convergence and convergence in law. As the last name suggests, the
convergence only depends on the induced laws of the vectors and not on the probability
spaces on which they are defined. Weak convergence is denoted by X n X; if X has dis-
tribution L, or a distribution with a standard code, such as N(O, 1), then also by X n L or
X n N(O, 1).
Let d (x, y) be a distance function on JRk that generates the usual topology. For instance,
the Euclidean distance
( k ) 1/2
d(x, y) = I!x - YI! = (Xi - Yi)2 .
A sequence of random variables X n is said to converge in probability to X if for all £ > O
P(d(X n , X) > £) --+ O.
p
This is denoted by X n --+ X. In this notation convergence in probability is the same as
p
d(Xn, X) --+ O.
t More formally it is a Borel measurable map from some probability space in IR. k . Throughout it is implic-
itly understood that variables X, g(X), and so forth of which we eompute expeetations or probabilities are
measurable maps on some probability space.
5
6
Stochastic Convergence
As we shall see, convergence in probability is stronger than convergence in distribution.
An even stronger mode of convergence is almost-sure convergence. The sequence X n is
said to converge almost surely to X if d (X n , X) ---+ O with probability one:
p (lim d (X n, X) == O) == 1.
This is denoted by X n X. Note that convergence in probability and convergence almost
surely only make sense if each of X n and X are defined on the same probability space. For
convergence in distribution this is not necessary.
2.1 Example (Classicallimit theorems). Let Y n be the average of the first n of a sequence
of independent, identically distributed random vectors YI, Y 2 , . . . . If Eli YI II < 00, then
Y n EY I by the strong law oflarge numbers. Underthe strongerassumption thatEli YI 11 2 <
00, the central limit theorem asserts that .Jn( Y n - EY I ) 'v'7 N(O, Cov YI). The central limit
theorem plays an important role in this manuscript. It is proved later in this chapter, first
for the case of real variables, and next it is extended to random vectors. The strong law
of large numbers appears to be of less interest in statistics. Usually the weak law of large
numbers, according to which Y n EY I , suffices. This is proved later in this chapter. D
The portmanteau lemma gives a number of equivalent descriptions of weak convergence.
Most of the characterizations are only useful in proofs. The last one also has intuitive value.
2.2 Lemma (Portmanteau). For any random vectors X n and X thefollowing statements
are equivalent.
(i) P(X n < x) ---+ P(X < x) for all continuity points of x r-+ P(X < x);
(ii) Ef(X n ) ---+ Ef(X)forall bounded, continuousfunctions f;
(iii) Ef(X n ) ---+ Ef(X) for all bounded, Lipschitz t functions f;
(iv) lim infEf (X n ) > Ef (X) for all nonnegative, continuous functions f;
(v) liminfP(Xn E G) > P(X E G)for every open set G;
(vi) lim sup P(X n E F) < P(X E F) for every closed set F;
(vii) P(X n E B) ---+ P(X E B) for all Borel sets B with P(X E 8 B) O, where
8B == B - B is the boundary of B.
Proof. (i)::::} (ii). Assume first that the distribution function of X is continuous. Then
condition (i) implies that P(X n E I) ---+ P(X E I) for every rectangle I. Choose a
sufficiently large, compact rectangle I with P(X t/:. I) < £. A continuous function f is
uniformly continuous on the compact set I. Thus there exists a partition I == U i Ii into
finitely many rectangles Ii such that f varies at most £ on every Ij. Take a point X j from
each Ii and define fe == Li f (x i) 1 Ij . Then I f - fel < £ on I, whence if f takes its values
in [ - 1, 1],
lE f (X n) - Efe (X n) I < £ + p (X n t/:. I),
IEf(X) - Efe(X) I < £ + P(X t/:. I) < 2£.
t A funetion is called Lipschitz if there exists a number L such that I f (x) - f (y) I ::s Ld (x, y), for every x and
y. The least such number L is denoted II f Ilhp.
2.1 Basic Theory
7
For sufficiently large n, the right side of the first equation is smaller than 28 as well. We
combine this with
IEfc(X n ) - Efc(X)I < l:lp(X n E I j ) - P(X E Ij)llf(xj)l-+ O.
j
Together with the triangle inequality the three displays show that IEf(X n ) - Ef(X) I is
bounded by 58 eventually. This being true for every 8 > O implies (ii).
Call a set B a continuity set if its boundary 8B satisfies P(X E 8B) == O. The preceding
argument is valid for a general X provided all rectangles I are chosen equal to continuity
sets. This is possible, because the collection of discontinuity sets is sparse. Given any
collection of pairwise disjoint measurable sets, at most countably many sets can have
positive probability. Otherwise the probability of their union would be infinite. Therefore,
given any collection of sets {Ba : ex E A} with pairwise disjoint boundaries, all except at
most countably many sets are continuity sets. In particular, for each j at most countably
many sets of the form {x : X j < ex} are not continuity sets. Conclude that there exist dense
subsets Q 1, . . . , Qk of JR such that each rectangle with corners in the set Q 1 X . . . X Qk is
a continuity set. We can choose all rectangles I inside this set.
(iii) ::::} (v). For every open set G there exists a sequence of Lipschitz functions with
O < tm t lG' For instance fm(x) == (md(x, G C )) /\ 1. For every fixed m,
liminfP(Xn E G) > liminfEfm(X n ) == Efm(X).
n ----+ 00 n ----+ 00
As m -+ 00 the right side increases to P(X E G) by the monotone convergence theorem.
(v) {} (vi). Because a set is open if and only if its complement is closed, this follows by
taking complements.
(v) + (vi) ::::} (vii). Let B and B denote the interior and the closure of a set, respectively.
By (iv)
o o
P(X E B) < liminfP(Xn E B) < lim sup P(X n E B) < P(X E B),
by (v). If P(X E 8B) == O, then left and right side are equal, whence all inequalities
are equalities. The probability P(X E B) and the limit limP(Xn E B) are between the
expressions on left and right and hence equal to the common value.
(vii) ::::} (i). Every cell (-00, x] such that x is a continuity point of x P(X < x) is a
continuity set.
The equivalence (ii) {} (iv) is left as an exercise. .
The continuous-mapping theorem is a simple result, but it is extremely usefu1. If the
sequence ofrandom vectors X n converges to X and g is continuous, then g(X n ) converges
to g (X). This is true for each of the three modes of stochastic convergence.
2.3 Theorem (Continuous mapping). Let g : JRk IR m be continuous at every point of
a set C such that P(X E C) == 1.
(i) If X n 'v'-t X, then g(X n ) -v-+ g(X);
(ii) If X n X, then g(Xn) g(X);
(iii) If X n X, then g(Xn) g(X).
8
Stochastic Convergence
Proof. (i). The event {g(X n ) EF} is identical to the event {X n E g-l (F) }. For every
closed set F,
g-l (F) C g-l (F) C g-l (F) U CC.
To see the second inclusion, take X in the closure of g-l (F). Thus, there exists a sequence
X m with X m ---+ X and g(xm) E F for every F. If X E C, then g(xm) ---+ g(x), which is in F
because F is closed; otherwise X E CC. By the portmanteau lemma,
limsupP(g(X n ) E F) < limsupP(X n E g-l(F)) < p(X E g-l(F)).
Because P(X E CC) == O, the probability on the right is p(X E g-l (F)) == p(g(X) E
F). Apply the portmanteau lemma again, in the opposite direction, to conclude that
g(Xn) g(X).
(ii). Fix arbitrary £ > O. For each 8 > O let Bo be the set of X for which there exists
y with d(x, y) < 8, but d(g(x), g(y)) > £. If X Bo and d(g(X n ), g(X)) > 8, then
d(Xn, X) > 8. Consequently,
P(d(g(X n ), g(X)) > e) < P(X E B 8 ) + P(d(X n , X) > 8).
The second term on the right converges to zero as n ---+ 00 for every fixed 8 > O. Because
Bo n C + eJ by continuity of g, the first term converges to zero as 8 + O.
Assertion (iii) is trivial. .
Any random vec tor X is tight: For every £ > O there exists a constant M such that
p(IIXII > M) < £. A set ofrandom vectors {X a : a E A} is called uniformly tightif M can
be chosen the same for every Xa: For every £ > O there exists a constant M such that
supp(IIXall > M) < £.
a
Thus, there exists a compact set to which all Xa give probability "almost" one. Another
name for uniformly tight is bounded in probability. It is not hard to see that every weakly
converging sequence X n is uniformly tight. More surprisingly, the converse of this statement
is almost true: According to Prohorov's theorem, every uniformly tight sequence contains a
weakly converging subsequence. Prohorov's theorem generalizes the Heine-Borel theorem
from deterministic sequences X n to random vectors.
2.4 Theorem (Prohorov's theorem). Let X n be random vectors in :IR k .
(i) If X n X for some X, then {X n : n E N} is uniformly tight;
(ii) If X n is uniformly tight, then there exists a subsequence with X nj X as j ---+ 00,
for some x.
Proof. (i). Fix a number M such that p(IIXII > M) < £. By the portmanteau lemma
p(11 X n II > M) exceeds p(11 X II > M) arbitrarily little for sufficiently large n. Thus there
exists N such that p(IIX n II > M) < 2£, for all n > N. Because each of the finitely many
variables X n with n < N is tight, the value of M can be increased, if necessary, to ensure
that p(11 X n II > M) < 2£ for every n.
2.1 Basic Theory
9
(ii). By Helly's lemma (described subsequent1y), there exists a subsequence Fnj of
the sequence of cumulative distribution functions Fn (x) == P(X n < x) that converges
weakly to a possibly "defective" distribution function F. It suffices to show that F is a
proper distribution function: F (x) ---+ O, 1 if Xi ---+ -00 for some i, or x ---+ 00. By the
uniform tightness, there exists M such that Fn (M) > 1- E for all n. By making M larger, if
necessary, it can be ensured that M is a continuity point of F. Then F(M) == lim Fnj (M) >
1 - 8. Conclude that F (x) ---+ 1 as x ---+ 00. That the limits at -00 are zero can be seen in
a similar manner. .
The crux of the proof of Prohorov's theorem is Helly's lemma. This asserts that any
given sequence of distribution functions contains a subsequence that converges weakly to
a possibly defective distribution function. A defective distribution function is a function
that has all the properties of a cumulative distribution function with the exception that it has
limit s less than 1 at (X) and/or greater than O at -00.
2.5 Lemma (Helly's lemma). Each given sequence Fn of cumulative distributionfunc-
tions on }Rk possesses a subsequence Fn j with the property that Fn j (x) ---+ F (x) at each
continuity point x of a possibly defective distribution function F.
Proof. Let Qk == {ql, q2, . . .} be the vectors with rational coordinates, ordered in an
arbitrary manner. Because the sequence Fn (ql) is contained in the interval [O, 1], it has
a converging subsequence. Call the indexing subsequence {n} } l and the limit G(ql).
Next, extract a further subsequence {nJ} C {n}} along which Fn (q2) converges to a
limit G(q2), a further subsequence {n]} c {nJ} along which F n (q3) converges to alimit
G (q3), . . . '. and so forth. The "tail" of the diagonal sequence nj: == nj belongs to every
sequence nj. Renee Fnj (qi) ---+ G(qi) for every i == 1,2, . . . . Because each Fn is nonde-
creasing, G(q) < G(q') if q < q'. Define
F(x) == inf G(q).
q>x
Then F is nondecreasing. It is also right-continuous at every point x, because for every
E > O there exists q > x with G(q) - F(x) < E, which implies F(y) - F(x) < E for every
x < y < q. Continuity of F at x implies, for every 8 > O, the existence of q < x < q'
such that G(q') - G(q) < E. By monotonicity, we have G(q) < F(x) < G(q'), and
G(q) == lim Fnj (q) < liminf Fnj (x) < lim Fnj (q') == G(q').
Conclude that Ilim inf Fn j (x) - F (x) I < E. Because this is true for every E > O and
the same result can be obtained for the lim sup, it follows that Fn j (x) ---+ F (x) at every
continuity point of F.
In the higher-dimensional case, it must still be shown that the expressions defining masses
of cells are nonnegative. For instance, for k == 2, F is a (defective) distribution function
only if F(b) + F(a) - F(al, b 2 ) - F(a2, bI) > O for every a < b. In the case that the four
corners a, b, (al, b 2 ), and (a2, bI) of the cell are continuity points; this is immediate from
the convergence of Fn j to F and the fact that each Fn is a distribution function. N ext, for
general cells the property follows by right continuity. .
10
Stochastic Convergence
2.6 Example (Markov's inequality). A sequence X n ofrandorn variables with EIXnlP ==
0(1) for some p > O is uniformly tight. This follows because by Markov's inequality
EIX IP
p(IX n [ > M) < M:
The right side can be made arbitrarily small, uniformly in n, by choosing sufficiently
large M.
Because EX == var X n + (EX n )2, an alternative sufficient condition for uniform tight-
ness is EX n == O (1) and var X n == O (1). This cannot be reversed. D
Consider some of the relationships among the three modes of convergence. Convergence
in distribution is weaker than convergence in probability, which is in turn weaker than
almost-sure convergence, except if the limit is constant.
2.7 Theorem. Let X n , X and Y n be random vectors. Then
(i) X n X implies X n X;
(ii) X n X implies X n 'v'7 X;
(iii) X n c for a constant c if and only if X n 'v'7 c;
(iv) if X n 'v'7 X and d (X n , Y n ) O, then Y n 'v'7 X;
(v) if X n 'v'7 X and Y n c for a constant c, then (X n , Y n ) 'v'7 (X, c);
(vi) if X n X and Y n Y, then (X n , Y n ) (X, Y).
Proof. (i). The sequence of sets An == Um n{d(Xm' X) > E} is decreasing for every
8 > O and decreases to the empty set if X n «())) -+ X «())) for every ()). If X n X, then
P(d(X n , X) > 8) < P(An) -+ O.
(iv). For every f with range [O, 1] and Lipschitz norm at most 1 and every 8 > O,
IEf(X n ) -Ef(Yn)1 < 8EI{d(X n , Y n ) < E} +2EI{d(X n , Y n ) > 8}.
The second term on the right converges to zero as n -+ 00. The first term can be rnade
arbitrarily small by choice of E. Conclude that the sequences Ef(X n ) and Ef(Y n ) have the
same limit. The result follows from the portmanteau lemma.
(ii). Because d(Xn, X) O and trivially X X, it follows that X n 'v'7 X by (iv).
(iii). The "only if" part is a special case of (ii). For the converse let ball(c, 8) be the open
ball of radius E around c. Then P(d(X n , c) > E) == p(X n E ball(c, 8)c). If X n 'v'7 c, then
the lim sup of the last probability is bounded by P( c E ball(c, E )c) == O, by the portmanteau
lemma.
(v). First note that d((X n , Y n ), (X n , c)) == d(Y n , c) O. Thus, according to (iv), it
suffices to show that (X n , c) 'v'7 (X, c). For every continuous, bounded function (x, y) r--+
f(x, y), the function x r--+ f(x, c) is continuousandbounded. ThusEf(X n , c) -+ Ef(X, c)
if X n X.
(vi). This follows from d(XI, YI), (X2, Y2)) < d(XI, X2) + d(YI, Y2). .
According to the last assertion of the lemma, convergence in probability of a sequence of
vectors X n == (X n , I, . . . , Xn,k) is equivalent to convergence of every one of the sequences
of components Xn,i separately. The analogous statement for convergence in distribution
2.1 Basic Theory
11
is false: Convergence in distribution of the sequence X n is stronger than convergence of
every one of the sequences of components Xn,i. The point is that the distribution of the
components Xn,i separately does not determine their joint distribution: They might be
independent or dependent in many ways. We speak of joint eonvergenee in distribution
versus marginal eonvergenee .
Assertion (v) of the lemma has some useful consequences. If X n X and Y n c, then
(X n , Y n ) (X, c). Consequently, by the continuous mappingtheorem, g(X n , Y n ) g(X, c)
for every map g that is continuous at every point in the set k x {e} in which the vector
(X, c) takes its values. Thus, for every g such that
lim g (x, y) == g (xo, c),
x xo,y c
for every xo.
Some particular applications of this principle are known as Slutsky's lemma.
2.8 Lemma (Slutsky). Let X n , X and Y n be random vectors orvariables. IfX n X and
Y n e for a constant e, then
(i) X n + Y n X + e;
(ii) YnX n eX;
(iii) y n - I X n 'v'7 c- I X provided e i= O.
In (i) the "constant" e must be a vector of the same dimension as X, and in (ii) it is
probably initially understood to be a sc al ar. However, (ii) is also true if every Y n and c
are matrices (which can be identified with vectors, for instance by aligning rows, to give a
meaning to the convergence Y n c), simply because matrix multiplication (x, y) 1--+ yx is
a continuous operation. Even (iii) is valid for matrices Y n and e and vectors X n provided
e i= O is understood as c being invertible, because taking an inverse is also continuous.
2.9 Example (t-statistic). Let YI, Y 2 , . . . be independent, identically distributed random
variables with EY I == O and EY? < 00. Then the t-statistic ..jn Y n / Sn, where S == (n -
1)-1 E7=I (Yi - Y n)2 is the sample variance, is asymptotically standard norma!.
To see this, first note that by two applications of the weak law of large numbers and the
continuous-mapping theorem for convergence in probability
2 n ( 1 2 -2 ) P ( 2 2 )
Sn == - Yi - Y n -+ 1 EY I - (EY I ) == varY I .
n - 1 n i=I
Again by the continuous-mapping theorem, Sn converges in probability to sd YI. By the cen-
trallimit theorem ..jn Y n converges in law to the N(O, var YI) distribution. Finally, Slutsky's
lemma gives that the sequence of t-statistics converges in distribution to N (O, var YI) / sd YI
== N(G, 1). O
2.10 Example (Con.fidence intervals). Let Tn and Sn be sequences of estimators satis-
fying
(Tn - e) N(O, a 2 ),
S2 a 2
n '
for certain parameters e and a 2 depending on the underlying distribution, for every distri-
bution in the model. Then e == Tn ::i:: Sn / ..jn Za is a confidence interval for e of asymptotic
12
Stochastic Convergence
level 1 - 2a. More precisely, we have that the probability that e is contained in ['01 -
Sn I ,jn Za, Tn + Sn I ,jn Za] converges to 1 - 2a.
This is a consequence of the fact that the sequence ,jn (Tn - e) I Sn is asymptotically
standard normally distributed. D
If the limit variable X has a continuous distribution function, then weak convergence
X n X implies P(X n < x) -+ P(X < x) for every x. The convergence is then even
uniform in x.
2.11 Lemma. Suppose that X n X for arandom vector X with a continuous distribution
function. Then sUPx Ip(X n < x) - P(X < x) I -+ O.
Proof. Let Fn and F be the distribution functions of X n and X. First consider the one-
dimensional case. Fix kEN. By the continuity of F there exist points -00 == Xo <
XI < . . . < Xk == 00 with F(Xi) == i / k. By monotonicity, we have, for Xi-I < X < Xi,
Fn(x) - F(x) < Fn(Xi) - F(Xi-I) == Fn(Xi) - F(Xi) + lik
> Fn(Xi-I) - F(Xi) == Fn(Xi-I) - F(Xi-I) -lik.
Thus I Fn (x) - F(x) I is bounded above by SUPi I Fn (Xi) - F(Xi) I + II k, for every x. The
latter, finite supremum converges to zero as n -+ 00, for each fixed k. Because k is arbitrary,
the result follows.
In the higher-dimensional case, we follow a similar argument but use hyperrectangles,
rather than intervals. We can construct the rectangles by intersecting the k partitions obtained
by subdividing each coordinate separately as before. .
2.2 Stochastic o and O Symbols
It is convenient to have short expressions for terms that converge in probability to zero
or are uniformly tight. The notation o p (1) ("small oh-P-one") is short for a sequence of
random vectors that converges to zero in probability. The expression O p (1) ("big oh-
P-one") denotes a sequence that is bounded in probability. More generally, for a given
sequence of random variables Rn,
X n == op(R n )
X n == Op(R n )
means
means
X n == YnRn
X n == YnRn
and
and
p
Y n -+ O;
Y n == Opel).
This expresses that the sequence X n converges in probability to zero or is bounded in
probability at the "rate" Rn. For deterministic sequences X n and Rn, the stochastic "oh"
symbols reduce to the usualoand O from calculus.
There are many rules of calculus with o and O symbols, which we apply without com-
ment. For instance,
op(l) + opel) == opel)
opel) + Op(l) == Opel)
Op(l)op(l) == opel)
2.3 Characteristic Functions
13
(1 +op(l))-l == Op(l)
o p (Rn) == Rno p (1)
Op(R n ) == R n O p (l)
op(Op(l)) == op(l).
To see the validity of these rules it suffices to restate them in terms of explicitly named
vectors, where each o p (1) and O p (1) should be replaced by a different sequence of vectors
that converges to zero or is bounded in probability. In this way the first rule says: If X n O
and Y n O, then Zn == X n + Y n O. This is an example of the continuous-mapping
theorem. The third rule is short for the following: If X n is bounded in probability and
Y n O, then XnY n O. If X n would also converge in distribution, then this would be
statement (ii) of Slutsky's lemma (with c == O). But by Prohorov's theorem, X n converges
in distribution "along subsequences" if it is bounded in probability, so that the third rule
can still be deduced from Slutsky's lemma by "arguing along subsequences."
N ote that both rules are in fact implications and should be read from left to right, even
though they are stated with the help of the equality sign. Similarly, a1though it is true that
o p (1) + o p (1) == 20 p (1), writing down this rule does not reflect understanding of the o p
symbol.
Two more complicated rules are given by the following lemma.
2.12 Lemma. Let R be afunction defined on domain in JRk such that R(O) == O. Let X n be
a sequence of random vectors with values in the domain of R that converges in probability
to zero. Then, for every p > O,
(i) if R(h) == o(llhII P ) as h O, then R(X n ) == op(IIXnllp);
(ii) if R(h) == O(lIhIl P ) as h O, then R(X n ) == Op(IIXnllp).
Proof. Define g (h) as g (h) == R (h) / II hI/P for h #- O and g (O) == O. Then R (X n ) ==
g(Xn) IIX n IIP.
(i) Because the function g is continuous at zero by assumption, g(X n ) g(O) == O by
the continuous-mapping theorem.
(ii) By assumption there exist M and o > O such that Ig(h) I < M whenever I/h II < o.
Thus p(lg(Xn)1 > M) < p(IIX n 1/ > o) O, and the sequence g(Xn) is tight. .
*2.3 Characteristic Functions
It is sometimes possible to show convergence in distribution of a sequence of random vectors
directly from the definition. In other cases "transforms" of probability measures may help.
The basic idea is that it suffices to show characterization (ii) of the portmanteau lernrna for
a srnall subset of functions f only.
The most irnportant transform is the characteristic function
t 1---+ Ee itT X
,
t E JRk.
Each of the functions X 1---+ eit T X is continuous and bounded. Thus, by the portrnanteau
lernrna, Ee itT X n Ee žtT X for every t if X n 'v'-t X. By Levy's continuity theorem the
14
Stochastic Convergence
converse is also true: Pointwise convergence of characteristic functions is equivalent to
weak convergence.
2.13 Theorem (Levy's continuity theorem). Let X n and X be random vectors in R k .
Then X n X if and only ifEeitTXn Ee itTX for every t E R k . Moreover, if EeitTXn con-
verges pointwise to a function ep (t) that is continuous at zero, then ep is the characteristic
function of arandom vector X and X n X.
Proof. If X n X, then Eh(X n ) Eh(X) for every bounded continuous function h, in
particular for the functions h (x) eit T x. This gives one direction of the first statement.
For the proof of the last statement, suppose first that we already know that the sequence
X n is uniformly tight. Then, according to Prohorov's theorem, every subsequence has a
further subsequence that converges in distribution to some vec tor Y. By the preceding
paragraph, the characteristic function of Y is the limit of the characteristic functions of the
converging subsequence. By assumption, this limit is the function ep (t). Conclude that
every weak limit point Y of a converging subsequence possesses characteristic function
ep. Because a characteristic function uniquely determines a distributian (see Lemma 2.15),
it follows that the sequence X n has only one weak limit point. It can be checked that a
uniformly tight sequence with a unique limit point converges to this limit point, and the
proof is comp1ete.
The uniform tightness of the sequence X n can be derived from the continuity of ep at
zero. Because marginal tightness implies joint tightness, it may be assumed without 10ss
of generality that X n is one-dimensional. For every x and 8 > O,
1{1 8x l > 2} < 2 ( 1 - Sin8X ) J 8 (1 - costx) dt.
8x 8-0
Replace x by X n , take expectations, and use Fubini's theorem to obtain that
P ( IXnl > ) < J 8 Re(I - Ee itXn ) dt.
8 8-0
By assumption, the integrand in the right side converges pointwise to Re (1 - ep (t) ). By the
dominated-convergence theorem, the whole expression converges to
J 8 Re(I - 4>(t)) dt.
8 -o
Because ep is continuous at zero, there exists for every £ > O a 8 > O such that 1 1 - ep (t) I < £
for Itl < 8. For this 8 the integral is bounded by 2£. Conclude that p(IX n I > 2/8) < 2£
for sufficiently large n, whence the sequence X n is uniformly tight. .
2.14 Example (Normal distribution). The characteristic function of the Nk (IL, b) distri-
bution is the function
ifT /-l-! t T bt
tl--+e 2
Indeed, if X is Nk (O, I) distributed and b 1/2 is a symmetric square root of b (hence
b (:E 1/2)2), then :E 1/2 X + IL possesses the given normal distribution and
E ZT ( bI/2X+" ) zTll f (bl/2Z)Tx-!xTx d 1 ZTf.l+!ZTbZ
e t'" e t'" e 2 x e 2 .
(2n )k/2
2.3 Characteristic Functions
15
For real-valued z, the last equality follows easily by completing the square in the exponent.
Evaluating the integral for complex z, such as z == it, requires some skill in complex
function theory. One method, which avoids further calculations, is to show that both the
left- and righthand sides of the preceding display are analytic functions of z. For the right
side this is obvious; for the left side we can justify differentiation under the expectation
sign by the dominated-convergence theorem. Because the two sides agree on the real axis,
they must agree on the complex plane by uniqueness of analytic continuation. O
2.15 Lemma. Random vectors X and Y in IR k are equal in distribution if and only if
E e it 1 X == Ee itTy for every t E IR k .
Proof. By Fubini' s theorem and calculations as in the preceding example, for every (j > O
and y E IR k ,
f e- itTy e- tTtcr2 Ee itTX dt = E f eitT(X-Y) e- tTtcr2 dt
(2 ) k/2 1
= Jr k Ee-z(X-y)T(x-y)/cr2,
(j
By the convolution formula for densities, the righthand side is (2n)k times the density
p X +0- Z (y) of the sum of X and a Z for a standard normal vec tor Z that is independent of X.
Conclude that if X and Y have the same characteristic function, then the vectors X + a Z
and Y + a Z have the same density and hence are equal in distribution for every a > O. By
Slutsky's lemma X + (j Z -v-+ X as (j {. O, and similarly for Y. Thus X and Y are equal in
distribution. .
The characteristic function of a sum of independent variables equals the product of the
characteristic functions of the individual variables. This observation, combined with Levy's
theorem, yields simple proofs of both the law oflarge numbers and the central limit theorem.
2.16 Proposition (Weak law of large numbers). Let YI, . . . , Y n be i.i.d. random variables
with characteristic function ep. Then Y n IL for a real number IL if and only if ep is dijfer-
entiable at zero with i IL == ep' (O).
Proof. We only prave that differentiability is sufficient. For the converse, see, for exam-
ple, [127, p. 52]. Because ep (O) == 1, differentiability of ep at zero means that ep(t) == 1
+ tep' (O) + o(t) as t -+ O. Thus, by Fubini's theorem, for each fixed t and n -+ 00,
Ee itY " = n( : ) = (1 + : ifL+O( ) r -+ e itM ,
The right side is the characteristic function of the constant variable IL. By Levy's theorem,
Y n converges in distribution to IL- Convergence in distribution to a constant is the same as
convergence in probability. .
A sufficient but not necessary condition for ep (t) == Ee itY to be differentiable at zero
is that El Y I < 00. In that case the dominated convergence theorem allows differentiation
16
Stochastic Convergence
under the expectation sign, and we obtain
d . Y . Y
q/ (t) == - Ee lt == Ei Ye lt .
dt
In particular, the derivative at zero is ep' (O) == iEY and hence Y n EY 1 .
If Ey 2 < 00, then the Taylor expansion can be carried a step further and we can obtain
a version of the central limit theorem.
2.17 Proposition (Central limit theorem). Let YI, . . . , Y n be i. i.d. random variables with
EY i == O and EY? == 1. Then the sequence ,Jri Y n converges in distribution to the standard
normal distribution.
Proof. A second differentiation under the expectation sign shows that ep" (O) == i 2 Ey2.
Because ep' (O) == iEY == O, we obtain
( 2 ) n
. - t It I 1 22
Ee ltJTiYn = cpn ( ,Jri ) = 1 - 2: -;;- Ey2 + o( n ) -+ e -'il EY
The right side is the characteristic function of the normal distribution with mean zero and
variance Ey 2 . The proposition follows from Levy's continuity theorem. .
The characteristic function t 1---+ Ee itT X of a vector X is determined by the set of all
characteristic functions u 1---+ Eeiu(tT X) of linear combinations t T X of the components of X.
Therefore, Levy' s continuity theorem implies that weak convergence of vectors is equivalent
to weak convergence of linear combinations:
X n X if and only if t T X n t T X for all t E k.
This is known as the Cramer- Wold device. It allows to reduce higher-dimensional problems
to the one-dimensional case.
2.18 Example (Multivariate central limit theorem). Let YI, Y 2 , . . . be i.i.d. random vec-
tors in k with mean vec tor f.-L == EY 1 and covariance matrix == E(Y 1 - f.-L)(Y I - f.-L)T.
Then
I n
,Jri L (Yi - f.-L) == ,Jn( Y n - f.-L) Nk(O, ).
n. 1
l=
(The sum is taken coordinatewise.) By the Cramer-Wold device, this can be proved by
finding the limit distribution of the sequences of real variables
T ( I ) I T T
t - L...,'cY ž - f.-L) == - (t Yi - t f.-L).
,Jrii=I ,Jri i=I
Because the random variables t T YI - t T f.-L, t T Y2 - t T f.-L, . . . are i.i.d. with zero mean and
variance t T t, this sequence is asymptotically NI (O, t T ht)-distributed by the univariate
central limit theorem. This is exactly the distribution of t T X if X possesses an NkCO, h)
distribution. O
2.5 Convergence of Moments
17
*2.4 Almost-Sure Representations
Convergence in distribution certainly does not imply convergence in probability or almost
sure1y. However, the following theorem shows that a given sequence X n -v-7 X can always
be replaced by a sequence i n -v-7 i that is, marginally, equa1 in distribution and converges
almost surely. This construction is sometimes useful and has been put to good use by some
authors, but we do not use it in this book.
2.19 Theorem (Almost-sure representations). Suppose that the sequence of random vec-
tors Kn converges in distribution to arandom vector Xo. Then there exists a probability
space (Q, U, P) and random vectors in defined on it such that in is equal in distribution
to X n for every n > O and in io almost surely.
Proof. For random variab1es we can simply define i n == F;; 1 (U) for Fn the distribu-
tion function of X n and U an arbitrary random variable with the uniform distribution on
[O, 1]. (The "quantile transformation," see Section 21.1.) The simp1est known construc-
tion for higher-dimensional vectors is more complicated. See, for example, Theorem 1.10.4
in [146], or [41]. .
*2.5 Convergence of Moments
By the portmanteau lemma, weak convergence X n -v-7 X implies that Ef (X n ) Ef (X) for
every continuous, bounded function f. The condition that f be bounded is not superfluous:
It is not difficult to find examples of a sequence X n -v-7 X and an unbounded, continuous
function f for which the convergence fails. In particular, in general convergence in distri-
bution does notimplyconvergenceEX EXP ofmoments. However, inmany situations
such convergence occurs, but it requires more ef fort to prove it.
A sequence of random variables Y n is called asymptotically uniformly integrable if
lim 1imsupEIY n l1{IY n l > M} == o.
M oo n oo
Uniform integrability is the missing link between convergence in distribution and conver-
gence of moments.
2.20 Theorem. Let f: JRk f---+ IR be measurable and continuous at every point in a set
C. Let X n -v-7 X where X takes its values in C. Then Ef(X n ) Ef(X) if and only ifthe
sequence ofrandom variables f(X n ) is asymptotically uniformly integrable.
Proof. We give the proof only in the most interesting direction. (See, for example, [146]
(p. 69) for the other direction.) Suppose that Y n == f (X n ) is asymptotically uniformly
integrable. Then we show that EY n EY for Y == f(X). Assume without 10ss of
generality that Y n is nonnegative; otherwise argue the positive and negative parts separately.
By the continuous mapping theorem, Y n -v-7 Y. By the triangle inequality,
IEY n - EYI < IEY n - EY n /\ MI + IEY n /\ M - EY /\ MI + \EY /\ M - EYI.
Because the function y f---+ Y /\ M is continuous and bounded on [O, (0), it follows that the
middle term on the right converges to zero as n 00. The first term is bounded above by
18
Stochastic Convergence
EY n 1 {Y n > M}, and converges to zero as n --* 00 followed by M --* 00, by the uniform
integrability. By the portmanteau lernma (iv), the third term is bounded by the lim inf as
n --* 00 of the first and hence converges to zero as M too. .
2.21 Example. Suppose X n is a sequence of random variables such that X n X and
limsupEIXnlP < 00 for some p. Then all moments of order strictly less than p converge
also: EX --* EX k for every k < p.
By the preceding theorem, it suffices to prove that the sequence X is asymptotically
uniformly integrable. By Markov's inequality
EIXnlkl{IXnlk > M} < M 1 -p/kEIX n IP.
The limit superior, as n --* 00 followed by M --* 00, of the right side is zero if k < p. D
The moment function p 1--+ EX p can be considered a transform of probability distributions,
just as can the characteristic function. In general, it is not a true transform in that it does
determine a distribution uniquely only under additional assumptions. If alimit distribution
is uniquely determined by its moments, this transform can still be used to establish weak
convergence.
2.22 Theorem. Let X n and X be random variables such that EX --* EXP < 00 for
every pEN. Ifthe distribution of X is uniquely determined by its moments, then X n X.
Proof. Because EX == 0(1), the sequence X n is uniformly tight, by Markov' s inequality.
By Prohorov's theorem, each subsequence has a further subsequence that converges weakly
to alimit Y. By the preceding example the moments of Y are the limit s of the moments
of the subsequence. Thus the moments of Y are identical to the moments of X. Because,
by assumption, there is only one distribution with this set of moments, X and Y are equal
in distribution. Conclude that every subsequence of X n has a further subsequence that
converges in distribution to X. This implies that the whole sequence converges to X. .
2.23 Example. The normal distribution is uniquely determined by its moments. (See, for
example, [123] or [133, p. 293].) Thus EX --* O for odd p and EX --* (p -1) (p - 3) . . . 1
for even p implies that X n N(O, 1). The converse is false. D
*2.6 Convergence-Determining Classes
A class F of functions f : JRk --* JR is called convergence-determining if for every sequence
of random vectors X n the convergence X n 'v'7 X is equivalent to Ef (X n ) --* Ef (X) for
every f E F. By definition the set of all bounded continuous functions is convergence-
determining, but so is the smaller set of all differentiable functions, and many other classes.
The set of all indicator functions 1 (-oo,t] would be convergence-determining if we would
restrict the definition to limits X with continuous distribution functions. We shall have
occasion to use the following results. (For proofs see Corollary 1.4.5 and Theorem 1.12.2,
for example, in [146].)
2.7 Law of the Iterated Logarithm
19
2.24 Lemma. On JRk == JRI X JRm the set offunctions (x, y) f(x)g(y) with f and g
ranging over all bounded, continuous functions on JRI and JRm, respectively, is convergence-
determining.
2.25 Lemma. There exists a countable set of continuous functions f : JRk [O, 1] that
is convergence-determining and, moreover, X n X implies that Ef(X n ) Ef(X) uni-
formly in f E :F.
*2.7 Law of the Iterated Logarithm
The law of the iterated logarithm is an intriguing result but appears to be of less interest
to statisticians. It can be viewed as a refinement of the strong law of large numbers.
If YI, Y2, . . . are i.i.d. random variables with mean zero, then YI + ... + Y n == o(n)
almo st surely b y the strong law. The law of the iterated logarithm improves this order to
O( ,J n log log n), and even gives the proportionality constant.
2.26 Proposition (Law of the iterated logarithm). Let YI, Y 2 , . . . be i.i.d. random vari-
ables with mean zero and variance 1. Then
. YI + ... + Y n
hm sup == v 2, a.s.
n--+oo ,J n log log n
Conversely, if this statement holds for both Yi and - Yi, then the variables have mean zero
and variance 1.
The law of the iterated logarithm gives an interesting illustration of the difference between
almost sure and distributional statements. Under the conditions of the proposition, the
sequence n- I / 2 (YI + . . . + Y n ) is asymptotically normally distributed by the central limit
theorem. The limiting no rmal dist ribution is spread out over the whole real line. Apparently
division by the factor ,J log log n is exactly right to keep n- I / 2 (Y I + ... + Y n ) within a
compact interval, eventually.
A simple application of Slutsky's lemma gives
YI + . . . + Y n p
Zn :== O.
,J n log log n
Thus Zn is with high probability contained in the interval ( -£, £) eventually, for any £ > O.
This appears to contradict the law of the iterated logarithm, which asserts that Zn reaches
the interval (-J2 - £, -J2 + £) infinitely often with probability one. The explanation is
that the set of w such that Zn(w) is in (-£, £) or (-J2 - £, -J2 + £) fluctuates with n. The
convergence in probability shows that at any advanced time a very large fraction of w have
Zn(w) E (-£, £). The law of the iterated logarithm shows that for each particular w the
sequence Zn (w) drops in and out of the interval (-J2 - £, -J2 + £) infinitely often (and
hence out of ( - £, £)).
The implications for statistics can be illustrated by considering confidence statements.
If IL and 1 are the true mean and variance of the sample YI, Y 2 , . . . , then the probability that
2 2
Y --<II<Y +-
n Jli- - n Jli
20
Stochastic Convergence
converges to <I:> (2) - <I:> (- 2) 95%. Thus the given interval is an asymptotic confidence
interval oflevel approximately 95%. (The confidence level is exactly <1>(2) - <P( -2) ifthe
observations are normally distributed. This may be assumed in the following; the accuracy
of the approximation is not an issue in this discussion.) The point J1- == O is contained in
the interval if and only if the variable 2n satisfies
2
1 2 nl < .
- -J log log n
Assume that J1- == O is the true value of the mean, and consider the following argument. By
the law of the iterated logarithm, we can be su re that 2n hits the interval (V2 - £, v2 + E)
infinitely often. The expression 2/ -J log log n is close to zero for large n. Thus we can be
sure that the true value I-L == O is outside the confidence interval infinitely often.
How can we solve the paradox that the usual confidence interval is wrong infinitel y often?
There appears to be a conceptual problem if it is imagined that a statistician collects data in
a sequential manner, computing a confidence interval for every n. However, although the
frequentist interpretation of a confidence interval is open to the usual criticism, the paradox
does not seem to rise within the frequentist framework. In fact, from a frequentist point
of view the curious conclusion is reasonable. Imagine 100 statisticians, all of whom set
95% confidence interval s in the usual manner. They all receive one observation per day
and update their confidence intervals daily. Then every day about five of them should have
a false interval. It is only fair that as the days go by all of them take turns in being unlucky,
and that the same five do not have it wrong all the time. This, indeed, happens according
to the law of the iterated logarithm.
The paradox may be partly caused by the feeling that with agrowing number of observa-
tions, the confidence interval s should become better. In contrast, the usual approach leads
to errors with certainty. However, this is only true if the usual approach is applied naively
in a sequential set-up. In practice one would do a genuine sequential analysis (including
the use of a stopping rule) or change the confidence level with n.
There is also another reason that the law of the iterated logarithm is of little practical
cons equence. The argument in the preceding paragraphs is based on the assumption that
2/ -J log log n is close to zero and is nonsensical if this quantity is larger than V2. Thus the
argument requires at least n > 1619, a respectable number of observations.
*2.8 Lindeberg-Feller Theorem
Central limit theorems are theorems concerning convergence in distribution of sums of
random variables. There are version s for dependent observations and nonnormal limit
distributions. The Lindeberg - Feller theorem is the simplest extension of the classical central
limit theorem and is applicable to independent observations with finite variances.
2.27 Proposition (Lindeberg-Feller central limit theorem). For each n let Y n ,l, . . . ,
Yn,k n be independent randoln vectors with finite variances such that
kn
LEIIYn.iI121{IIYn,ill > E} O,
i=l
every 8 > O,
kn
L COV Yn,i z=.
i=l
2.8 Lindeberg-Feller Theorem
21
Then the sequence L :l (Yn,i - EYn,i) converges in distribution to a normal N (O, h)
distribution.
A result of this type is necessary to treat the asymptotics of, for instance, regression
problerns with fixed covariates. We illustrate this by the line ar regression model. The
application is straightforward but notationalIy a bit involved. Therefore, at other places
in the rnanuscript we find it more convenient to assume that the covariates are arandom
sample, so that the ordinary central limit theorem applies.
2.28 Example (Linear regression). In the linear regression problem, we observe a vector
Y == Xf3 + e for a known (n x p) matrix X of fulI rank, and an (unobserved) error vector e
with i.i.d. components with mean zero and variance (J2. The least squares estimator of tJ is
== (X T X)-l X T y.
This estimator is unbiased and has covariance matrix (J2(XT X)-l. If the error vec tor e is
normally distributed, then is exactly normally distributed. Under reasonable conditions
on the design matrix, the least squares estimator is asymptoticalIy normalIy distributed for
a large range of error distributions. Here we fix p and let n tend to infinity.
This follows from the representation
n
(X T X)1/2( - f3) == (X T X)-1/2 X T e == Laniei,
i=l
where anI, . . . , ann are the columns of the (p x n) matrix (X T X)-1/2 X T ==: A. This sequence
is asymptoticalIy normal if the vectors anI el, . . . , anne n satisfy the Lindeberg conditions.
The norming matrix (X T X) 1/2 has been chosen to ensure that the vectors in the display
have covariance matrix (J2 I for every n. The remaining condition is
n
LIIani 112Eell {Ilani Illei I > £} -+ O.
i=l
This can be simplified to otherconditions in several ways. Because L !lani 11 2 == trace(AA T)
== p, it suffices that max Eef 1 {Ilani Illei I > £} -+ O, which is equivalent to
max Ilani II -+ O.
l i n
Alternatively, the expectation Ee 2 1{alel > £} can be bounded by £-kElel k + 2 a k and a
second set of sufficient conditions is
n
LIIani lik -+ O;
i=l
Ele11k < 00,
(k > 2).
Both sets of conditions are reasonable. Consider for instance the simple linear regression
model Yi == f30 + tJ1 X i + ei. Then
1 ( 1
(X T X)-I/2 X T==_ _
X
X ) -1/2 ( 1
x2 XI
1
:J.
X2
It is reasonable to assume that the sequences X and x2 are bounded. Then the first matrix
22
Stochastic Convergence
on the right behaves like a fixed matrix, and the conditions for asymptotic normality
simplify to
max IXi I == 0(n 1 / 2 ); or n 1 - k / 2 Ixlk ---+ O,
l:::i:::n
Elellk < 00.
Every reasonable design satisfies these conditions. O
*2.9 Convergence in Total Variation
A sequence of random variables converges in total variation to a variable X if
suplp(X n E B) - P(X E B) I ---+ O,
B
where the supremum is taken over all measurable sets B. In view of the portmanteau lemma,
this type of convergence is stronger than convergence in distribution. Not only is it required
that the sequence P(X n E B) converges for every Borel set B, the convergence must also
be uniform in B. Such strong convergence occurs less frequently and is often more than
necessary, whence the concept is less usefu1.
A simple sufficient condition for convergence in total variation is pointwise convergence
of densities. If X n and X have densities Pn and p with respect to a measure JL, then
suplp(X n E B) - P(X E B)I = J IPn - pI dJ1-.
B 2
Thus, convergence in total variation can be established by convergence theorems for inte-
grals from measure the ory. The following proposition, which should be compared with the
monotone and dominated convergence theorems, is most appropriate.
2.29 Proposition. Suppose that in and i are arbitrary measurable functions such that
in ---+ i JL-almost everywhere (or in JL-measure) and lim sup J linl P dJL < J lil P dJ1- <
00, ior some p > 1 and measure J1-. Then f I In - I I p d f.1, ---+ o.
Proof. By the inequality (a + b)P < 2 P a P + 2 P b P , valid for every a, b > O, and the
assumption, O < 2Pllnl P + 2 P I/IP - lin - IIP ---+ 2 P + 1 1/1 P almost everywhere. By
Fatou's lemma,
J 2 P + 1 1f1 P df.1, < liminf J (2 P lfnl P + 2 P lfl P -Ifn - fl P ) df.1,
< 2 P + 1 J Ifl P df.1, -lim sup J Ifn - fl P df.1"
by assumption. The proposition follows. .
2.30 Corollary (Scheffe). Let X n and X be random vectors with densities Pn and p with
respect to a measure J1-. If Pn ---+ p f.L-almost everywhere, then the sequence X n converges
to X in total variation.
Notes
23
The central limit theorem is usually formulated in term s of convergence in distribution.
Often it is valid in terms of the total variation distance, in the sense that
j I l(x nJ.L)2jna- 2
sup P(Y 1 + . . . + Y n E B) - -J2ii e - 2 - dx -+ O.
B B -Jna 2n
Here M and a 2 are mean and variance of the Yi, and the supremum is taken over all Borel
sets. An integrable characteristic function, in addition to a finite second moment, suffices.
2.31 Theorem (Central limit theorem in total variation). Let YI, Y 2 , . . . be i.i.d. random
variables with finite second moment and characteristic function ep such that J I ep (t) IV dt <
00 for some v > 1. Then YI + . . . + Y n satisfies the central limit theorem in total variation.
Proof. It can be assumed without loss of generality that EY I == O and var YI == 1. By
the inversion formula for characteristic functions (see [47, p. 509]), the density Pn of
YI + . . . + Y n / -Jn can be written
1 f . ( t ) n
Pn(x) == - e- ltx ep - dt.
2n -Jn
By the central limit theorem and Levy's continuity theorem, the integrand converges to
e- itx exp( _ t2). It will be shown that the integral converges to
1x 2
1 f . _1 t 2 e - 2
2n: e- ltX e 2 dt = -J2ii '
Then an application of Scheffe' s theorem concludes the pro of.
The integral can be split into two parts. First, for every 8 > O,
r e- itx <P ( ) n dt < .fii supl<p(t)l n - v f lep(t)lv dt.
J 1 tl>8,Jii v n Itl>8
Here sUPltl>8lep (t) I < 1 by the Riemann-Lebesgue lemma and because ep is the characteristic
function of a nonlattice distribution (e.g., [47, pp. 501, 513]). Thus, the first part of the
integral converges to zero geometrically fast.
Second, a Taylor expansion yields that ep(t) == 1 - t2 + 0(t 2 ) as t ---+ O, so that there
exists 8 > O such that I ep (t) I < 1 - t 2 /4 for every I ti < 8. It follows that
. ( t ) n ( t2 ) n
e- ltX <p -Jn 1{ Itl < 8.fii} < 1 - 4n
< e- t2j4 .
The proof can be concluded by applying the dominated convergence theorem to the remain-
ing part of the integral. .
Notes
The results of this chapter can be found in many introductions to probability theory. A
standard reference for weak convergence theory is the first chapter of [11]. Another very
readable introduction is [41]. The theory of this chapter is extended to random elements
with values in general metric spaces in Chapter 18.
24
Stochastic Convergence
PROBLEMS
1. If X n possesses a t-distribution with n degrees of freedom, then X n N(O, 1) as n -+ 00.
Show this.
2. Does it follow immediately from the resu1t of the previous exercise that Exf -+ EN (O, l)P for
every p E N? Is this true?
3. If X n N (O, 1) and Y n (5, then X n Y n N (O, (52). Show this.
4. In what sense is a chi-square distribution with n degrees of free dom approximately a normal
di s tributi on ?
5. Find an example of sequences such that X n X and Y n Y, but the joint sequence (X n , Y n )
does not converge in law.
6. If X n and Y n are independent random vectors for every n, then X n X and Y n Y imply that
(X n , Y n ) (X, Y), where X and Y are independent. Show this.
7. If every X n and X possess discrete distributions supported on the integers, then X n X if and
only ifP(X n == x) -+ P(X == x) for every integer x. Show this.
8. If P(X n == i / n) == 1/ n for every i == 1, 2, . . . , n, then X n X, but there exist Borel sets with
P(X n E B) == 1 for every n, but P(X E B) == O. Show this.
9. If P(X n == X n ) == 1 for numbers X n and X n -+ x, then X n x. Prove this
(i) by considering distributions functions
(ii) by using Theorem 2.7.
10. State the rule o p (1) + O p (1) == O p (1) in terms of random vectors and show its validity.
11. In what sense is it true that op(l) == Op(l)? Is it true that Op(l) == op(l)?
12. The rules given by Lemma 2.12 are not simple plug-in rules.
(i) Give an example of a function R with R(h) == o(llh \I) as h -+ O and a sequence of random
variables X n such that R(X n ) is not equal to o p (X n ).
(ii) Given an example of a function R such R (h) == O (1\ hil) as h -+ O and a sequence of random
variables X n such that X n == O p (1) but R(X n ) is not equal to O p (X n ).
13. Find an example of a sequence of random variables such that X n O, but EX n -+ 00.
14. Find an example of a sequence of random variables such that X n O, but X n does not converge
almost surely.
15. Let XI,..., X n be i.i.d. with density fA,a(x) == Ae- A (x-a)l{x > a}. Calculate the maximum
likelihood estimator of ( n, an) of (A, a) and show that ( n, an) (A, a).
16. Let XI, . . . , X n be i.i.d. standard normal variables. Show that the vector U == (X 1, . . . , X n ) / N,
where N 2 == L7=1 Xf, is uniformly distributed over the unit sphere sn-l in JRn, in the sense that
U and O U are identically distributed for every orthogonal transformation O of n .
17. For each n, let Un be uniformly distributed over the unit sphere sn-l in JRn. Show that the vectors
,.jn(U n ,l, U n ,2) converge in distribution to a pair of independent standard normal variables.
18. If ,.jn(T n - e) converges in distribution, then Tn converges in probability to e. Show this.
19. If EX n -+ fL and var X n -+ O, then X n fLo Show this.
20. If L l p(IX n I > 8) < 00 for every 8 > O, then X n converges almost surely to zero. Show this.
21. Use characteristic functions to show that binomial(n, A/n) Poisson(A). Why does the central
limit theorem not hold?
22. If XI, . . . , X n are i.i.d. standard Cauchy, then X n is standard Cauchy.
(i) Show this by using characteristic functions
(ii) Why does the weak law not hold?
23. Let XI, . . . , X n be i.i.d. with finite fourth moment. Find constants a, b, and c n such that the
sequence c n ( X n - a, X - b) converges in distribution, and determine the limit law. Here X n
and X are the averages of the Xi and the Xf, respectively.
3
Delta Method
The delta method eonsists of using a Taylor expansion to approximate a
random vector oftheform ep CTn) by the polynomial epce) + ep/ce)CT n -
e) + . . . in Tn - e. It is a simple but useful method to deduee the limit law
of ep CTn) - ep ce) from that ofTn - e. Applications inelude the nonrobust-
ness of the ehi-square test for normal variances and varianee stabilizing
transformations.
3.1 Basic Result
Suppose an estimator Tn for a parameter e is available, but the quantity of interest is ep ce) for
some known function ep. A natural estimator is ep CTn). How do the asymptotic properties
of ep CTn) follow from those of Tn ?
A first result is an immediate consequence of the continuous- mapping theorem. If the
sequence Tn converges in probability to e and ep is continuous at e, then ep CTn) converges
in probability to ep ce).
Of greater interest is a similar question conceming limit distributions. In particular, if
,Jn CTn - e) converges weakly to alimit distribution, is the same true for ,Jn( ep CTn) - ep ce))?
If ep is differentiable, then the answer is affirmative. Informally, we have
(ep CTn) - ep ce)) epl ce) CTn - e).
If ,Jn CTn - e) T for some variable T, then we expect that ,Jn (ep CTn) - ep ce) ) epl ce) T.
In p articul ar, if ,JnCT n - e) is asymptotically normal NCO, ( 2 ), then we expect that
,Jn(epCT n ) - epce)) is asymptotically normal N(O, ep / Ce)2a 2 ). This is proved in greater
generality in the following theorem.
In the preceding paragraph it is silently understood that Tn is real-valued, but we are more
interested in considering statistics ep CTn) that are formed out of several more basic statistics.
Consider the situation that Tn == CT n ,1, . . . , Tn,k) is vector-valued, and that ep :}Rk }Rm is
a given function defined at least on a neighbourhood of e. Recall that ep is differentiable at
e if there exists a linear map Cmatrix) ep :}Rk }Rm such that
epce + h) - epC()) == ep Ch) + o(llhll),
h O.
All the expressions in this equation are vectors of length m, and Ilh II is the Euclidean
nOffil. The line ar map h ep Ch) is sometimes called a "total derivative," as opposed to
25
26
Delta Method
partial derivatives. A sufficient condition for ep to be (totally) differentiable is that all partial
derivatives Bep j (x) / aXi exist for X in a neighborhood of 8 and are continuous at e. (Just
existence of the p arti al derivatives is not enough.) In any case, the total derivative is found
from the partial derivatives. If ep is differentiable, then it is partially differentiable, and the
derivative map h ep (h) is matrix multiplication by the matrix
acpl (e)
aXl
acpl (e)
aXk
ep ==
acpm (e)
aXl
acpm (e)
aXk
If the dependence of the derivative ep on e is continuous, then ep is called continuously
difJerentiable.
It is better to think of a derivative as a linear approximation h CjJ (h) to the function
h ep (e + h) - ep (e) than as a set of partial derivatives . Thus the derivative at a point 8
is a linear map. If the range space of ep is the real line (so that the derivative is a horizontal
vector), then the derivative is also called the gradient of the function.
Note that what is usually called the derivative of a function ep : JR JR does not com-
pletely correspond to the present derivative. The derivative at a point, usually written ep' (e),
is written here as ep . Although ep' (e) is a number, the second object ep is identified with the
map h ep (h) == ep' (e) h. Thus in the present terminology the usuaI derivative function
e ep' (8) is a map from IR into the set of linear maps from JR JR, not a map from
IR JR. Graphically the "affine" approximation h ep(e) + ep (h) is the tangent to the
function ep at e.
3.1 Theorem. Let ep : IlJ)cp C JRk ]Rm be a map defined on a subset of JRk and dif-
ferentiable at 8. Let Tn be random vectors taking their values in the domain of ep. If
r n (Tn - e) T for numbers r n 00, then r n (ep (Tn) - ep(e)) ep (T). Moreover, the
difJerence between rn(ep(Tn) - ep (e)) and ep (rn(Tn - e)) converges to zero inprobability.
Proof. Because the sequence r n (Tn - e) converges in distribution, it is uniformly tight and
Tn - e converges to zero in probability. By the differentiability of ep the remainder function
R(h) == ep(e + h) - ep(e) - ep (h) satisfies R(h) == o(llhll) as h O. Lemma 2.12 allows
to replace the fixed h by arandom sequence and gives
ep(Tn) - ep(e) - ep (Tn - e) - R(T n - e) == op(IIT n - eli).
Multiply this left and right with r n, and note that o p (r n II Tn - eli) == O P (1) by tightness of
the sequence r n (Tn - e). This yields the last statement of the theorem. Because matrix
multiplication is continuous, ep (r n (Tn - e) ) ep (T) by the continuous-mapping theorem.
Apply Slutsky's lemma to conclude that the sequence rn(ep(Tn) - ep(e)) has the same weak
limi 1. .
A common situation is that ,Jn(T n - e) converges to a multivariate normal distribution
Nk(J.L, :E). Then the conclusion of the theorem is that the sequence ,Jn(ep(Tn) - ep (e))
converges in law to the N m ( ep IL, ep :E (ep ) T) distribution.
3.2 Example (Sample variance). The sample variance of n observations XI, ..., X n
is defined as S2 == n-I :L7=1 (Xi - X )2 and can be written as ep e X , X2 ) for the function
3.1 Basic Result
27
ep (x, y) == y - x 2 . (For simplicity of notation, we divide by n rather than n -1.) Suppose that
S2 is based on a sample from a distribution with finite first to fourth moments al, a2, a3, a4.
By the mu1tivariate central limit theorem,
(( X ) ( al )) N (( O ) ( a2 - ai
n X2 - a2 "v-7 2 O' a3 - al a2
a3 -al 2 )) .
a4 - a 2
The map <p is differentiable at the point () == (al, a2)T, with derivative ep ( I a a ) == (-2al, 1).
1, 2
Thus if the vector (TI, T 2 r possesses the normal distribution in the last display, then
(ep( X , X2 ) - ep(al, a2)) "v-7 -2a I T I + T 2 .
The latter variable is normally distributed with zero mean and a varianee that can be ex-
pressed in al, . . . , a4. In case al == O, this varianee is simply a4 - ai. The general case
can be redueed to this ease, beeause S2 does not ehange if the observations Xi are replaeed
by the centered variables Yi == Xi - al. Write I1-k == EYt for the central moments of the
Xi. Noting that S2 == <p ( Y , y2 ) and that ep (11-1, 11-2) == 11-2 is the variance of the original
observations, we obtain
(S2 - JL2) "V-7N(O, JL4 - JL ).
In view of Slutsky's lemma, the same resu1t is valid for the unbiased version ni (n - 1)S2
of the sample variance, because ..;n(nl(n - 1) - 1) O. O
3.3 Example (Level of the ehi-square test). As an applieation of the preeeding example,
eonsider the ehi-square test for testing varianee. Normal the ory preseribes to reject the null
hypothesis Ha : JL2 < 1 for values of nS 2 exceeding the upper a point X;,a of the X;-l
distribution. If the observations are sampled from a normal distribution, then the test has
exaetly level a. Is this still approximately the ease if the underlying distribution is not
normal? Unfortunately, the answer is negative.
For large values of n, this can be seen with the help of the preeeding result. The central
limit theorem and the preceding example yield the two statements
X 2 - ( n - 1 )
n-I "v-7 N ( O 1 )
y' 2n - 2 ' ,
( : -1) N(O'K+2),
where K == JL41 JL - 3 is the ku rtosis o f the underlying distribution. The first statement
implies that (X;,a - (n - 1)) I yl 2n - 2) eonverges to the upper a point Za of the standard
normal distribution. Thus the level of the ehi -square test satisfies
( 2 2 ) ( C ( S2 ) X;,a - n ) ( za-V2 )
p 1Lz=1 nS > X n a = p v n - - 1 > ..;n -+ 1 - <I> .
, JL2 n YI K + 2
The asymptotie level reduces to 1 - <I> (za) == a if and only ifthe kurtosis of the underlying
distribution is O. This is the ease for normal distributions. On the other hand, heavy-tailed
distributions have a much larger kurtosis. If the kurtosis of the underlying distribution is
"close to" infinity, then the asymptotic level is elose to 1 - <t> (O) == 1/2. We eonclude that
the level of the ehi -square test is nonrobust against departures of normality that affeet the
value of the kurtosis. At least this is true if the eritieal values of the test are taken from
the chi -square distribution with (n - 1) degrees of freedom. If, instead, we would use a
28
Delta Method
Table 3.1. Level of the test that rejects
ifnS 2 / fJ.-2 exceeds the 0.95 quantile
of the X f9 distribution.
L L cl
Laplace 0.12
0.95 N (O, 1) + 0.05 N (0,9) 0.12
Note: Approximations based on simulation of
10,000 samples.
normal approximation to the distribution of ,Jn(S2 / f..L2 - 1) the problem would not arise,
provided the asymptotic variance K + 2 is estimated accurately. Table 3.1 gives the level
for two distributions with slight1y heavier tails than the normal distribution. D
In the preceding example the asymptotic distribution of ,Jn(S2 _(52) was obtained by the
delta method. Actually, it can also and more easily be derived by a direct expansion. Write
C 2 2 C ( 1 2 2 ) C - 2
yn(S -(5 )==yn - (Xi-f..L) -(5 -yn(X-f..L).
n. I
l=
The second term converges to zero in probability; the first term is asymptotically normal
by the central limit theorem. The whole expression is asymptotically normal by Slutsky's
lemma.
Thus it is not always a good idea to apply general theorems. However, in many exam-
ples the delta method is a good way to package the mechanics of Taylor expansions in a
transparent way.
3.4 Example. Consider the joint limit distribution of the sample variance S2 and the
t -statistic X / S. Again for the limit distribution it does not make a difference whether we
use a factor n or n-I to standardize S2. For simplicity we use n. Then (S2, X / S) can be
written as ep ( X , X2 ) for the map ep :}R2 r-+ }R2 given by
cfJ(x, y) = (y - x 2 , (y _ :2)1/2 ).
The joint limit distribution of ,Jn( X -al, X2 -(2) is derivedin the preceding example. The
map ep is differentiable at () == (al, (2) provided (52 == a2 - ai is positive, with derivative
( -2al
A.' - 2
'f'(al,a2) - al I
(a2-af)3/2 + (a2- a r)I/2
1 )
-al .
2(a2 -af)3/2
It follows that the sequence ,Jn (S2 - (52, X / S - al / (5) is asymptotically bivariate normally
distributed, with zero mean and covariance matrix,
( 2
ep' a2 - al
(al,a2) a3 - al a2
a3 - a1(2 ) (ep' )T
a4 - a (al,a2)'
It is easy but uninteresting to compute this explicit1y. D
3.1 Basic Result
29
3.5 Example (Skewness). The sample skewness of a sample XI, . . . , X n is defined as
n-I ( X. - X ) 3
I - L l =1 l
n- .
(n- 1 L:7=1 (Xi - X )2)3/2
Not surprisingly it converges in probability to the skewness of the underlying distribution,
defined as the quotient A == J..l3 / a 3 of the third central moment and the third power of the
standard deviation of one observation. The skewness of a symmetric distribution, such
as the normal distribution, equals zero, and the sample skewness may be used to test this
aspect of normality of the underlying distribution. For large samples a critical value may
be determined from the normal approximation for the sample skewness.
The sample skewness can be written as ep ( X , X2 , X3 ) for the function ep given by
c - 3ab + 2a 3
ep (a, b, c) == 2 3/2 .
(b -a )
The sequence ,Jn( X - al, X2 - a2, X3 - a3) is asymptotically mean-zero normal by the
central limit theorem, provided EX is finite. The value ep (al, a2, a3) is exactly the popu-
lation skewness. The function ep is differentiable at the point (al, a2, (3) and application of
the delta method is straightforward. We can save work by noting that the sample skewness
is location and scale invariant. With Yi == (Xi - a1)/a, the skewness can also be written as
ep ( Y , y2 , y3 ). With A == /-L3/ a 3 denoting the skewness of the underlying distribution, the
Y s satisfy
Y
,Jn y2 - 1
y3 - A
N ( O, (
K+3
A
K+2
J..ls / aS - A
K + 3 ))
/-Ls/as - A .
J..l6/ a 6 - A 2
The derivative of ep at the point (O, 1, A) equals (-3, -3A/2, 1). Renee, if T possesses the
normal distribution in the display, then ,Jn (in - A) is asymptotically normal distributed with
mean zero and variance equal to var( -3T 1 - 3AT 2 /2 + T 3 ). If the underlying distribution
is normal, then A == /-Ls == O, K == O and J..l6/ a 6 == 15. In that case the sample skewness is
asymptotically N (O, 6)-distributed.
An approximate level a test for normality based on the sample skewness could be to
reject normality if ,Jnlln I > ,j6 Za/2' Table 3.2 gives the level of this test for different
values of n. D
Table 3.2. Level of the test that
rejects if -Jnlln 1/,J6 exceeds the
0.975 quantile of the normal
distribution, in the case that the
observations are normally
distributed.
n Level
10 0.02
20 0.03
30 0.03
50 0.05
Note: Approximations based on simula-
tion of 10,000 samples.
30
Delta Method
3.2 Variance-Stabilizing Transformations
Given a sequence of statistics Tn with ,Jn(T n - e) N(O, a 2 (e)) for a range ofvalues of
e, asymptotic confidence interval s for e are given by
( a(e) a(e) )
Tn - Za ,Jn ' Tn + Za ,Jn .
These are asymptotically of level 1 - 2a in that the probability that e is covered by the
interval converges to 1- 2a for every e. Unfortunately, as stated previously, these interval s
are useless, because of their dependence on the unknown e. One solution is to replace
the unknown standard deviations a (e) by estimators. If the sequence of estimators is
chosen consistent, then the resulting confidence interval still has asymptotic level 1 - 2a.
Another approach is to use a variance-stabilizing transformation, which often leads to a
better approximation.
The idea is that no problem arises if the asymptotic variances a 2 (e) are independent of e .
Although this fortunate situation is rare, it is often possible to transform the parameter into
a different parameter rJ == ep (e), for which this idea can be applied. The natural estimator
for rJ is ep (Tn). If ep is differentiable, then
,Jn (ep (Tn) - ep (e)) N (O, ep' (e)2a 2 (e)).
For ep chosen such that ep' (e)a (e) - 1, the asymptotic variance is constant and finding an
asymptotic confidence interval for rJ == ep (e) is easy. The solution
ep(e) == f de
a (e)
is a variance-stabililizing transformation. variance stabililizing transformation. If it is
well defined, then it is automatically monotone, so that a confidence interval for rJ can be
transformed back into a confidence interval for e .
3.6 Example (Correlation). Let (XI, YI),..., (X n , Y n ) beasamplefromabivariatenor-
mal distribution with correlation coefficient p. The sample correlation coefficient is defined
as
r n ==
I: 7= 1 (Xi - X)(Y i - y)
{I:7=1 (Xi - X )2 I:(Y i - Y )2} 1/2'
With the help of the delta method, it is possible to derive that ,Jn (r - p) is asymptotically
zero-mean normal, with variance depending on the (mixed) third and fourth moment s of
(X, Y). This is true for general underlying distributions, provided the fourth moments exist.
Under the normality assumption the asymptotic variance can be expressed in the correlation
of X and Y. Tedious algebra gives
,Jn(r n - p) N(O, (1 _ p2)2).
It does not work very well to base an asymptotic confidence interval directly on this result.
3.3 Higher-Order Expansions
31
Table 3.3. Coverage probability of the asymptotic 95%
confidence interval for the correlation coefficient, for two
values of n and five dijferent values of the true correlation p.
n p == O P == 0.2
p == 0.4 p == 0.6 p == 0.8
0.92 0.93 0.92
0.94 0.94 0.94
15 0.92 0.92
25 0.93 0.94
Note: Approximations based on simulation of 10,000 samples.
cl
li)
-
cl
cl
(\j
cl
cl
-
cl
cl
-
cl
li)
cl
li)
cl
cl
0.0 0.2 0.4 O.S 0.8
-1.4 -1.0 -o. S -0.2
Figure 3.1. Histogram of 1000 sample correlation coefficients, based on 1000 independent samples
of the the bivariate normal distribution with correlation 0.6, and histogram of the arctanh of these
values.
The transformation
f 1 1 I+p
4J (p) == 2 dp == - log == arctanh p
I-p 2 I-p
is variance stabilizing. Thus, the sequence ,Jn(arctanh r-arctanh p) converges to a standard
normal distribution for every p. This leads to the asymptotic confidence interval for the
correlation coefficient p given by
( tanh (arctanh r - Za / -Jn), tanh (arctanh r + Za / -Jn) ) .
Table 3.3 gives an indication of the accuracy of this interval. Besides stabilizing the vari-
ance the arctanh transformation has the benefit of symmetrizing the distribution of the
sample correlation coefficient (which is perhaps of greater importance), as can be seen in
Figure 5.3. D
*3.3 Higher-Order Expansions
To package a simple idea in a theorem has the danger of obscuring the idea. The delta
method is based on a Taylor expansion of order one. Sometimes a problem cannot be
exactly forced into the framework described by the theorem, but the principle of a Taylor
expansion is stiH valid.
32
Delta Method
In the one-dimensional case, a Taylor expansion applied to a statistic Tn has the form
ep CTn) == ep ce) + CTn - e)ep/ ce) + CTn - e)2ep// ce) + . . . .
Usually the linear term CTn - e)ep/ ce) is of higher order than the remainder, and thus
determines the order at which ep CTn) - ep ce) converges to zero: the same order as Tn - e.
Then the approach of the preceding section gives the limit distribution of ep CTn) - epce). If
ep/ ce) == O, this approach is still valid but not of much interest, because the resulting limit
distribution is degenerate at zero. Then it is more informative to multiply the difference
ep CTn) - ep ce) by a higher rate and obtain a nondegenerate limit distribution. Looking at
the Taylor expansion, we see that the linear term disappears if ep/ ce) == O, and we expect
that the quadratic term determines the limit behavior of ep CTn).
3.7 Example. Suppose that .Jli x converges weakly to a standard normal distribution.
Because the derivative of X 1---+ cos X is zero at X == O, the standard delta method of the
preceding section yields that .JliCcos x - cos O) converges weakly to O. It should be
concluded that .Jli is not the right norming rate for the random sequence cos X - 1. A
more informative statement is that -2nCcos X -1) converges in distribution to a ehi-square
distribution with one degree of freedom. The explanation is that
- - 1- 2
cosX - cosO == CX - 0)0 +2:CX - O) Ccosx)I =O +....
That the remainder term is negligible after multiplication with n can be shown along the
same lines as the proof of Theorem 3.1. The sequence n X 2 converges in law to a xl
distribution by the continuous-mapping theorem; the sequence -2nCcos X - 1) has the
same limit, by Slutsky's lemma. D
Amore complicated situation arises if the statistic Tn is higher-dimensional with coor-
dinates of different orders of magnitude. For instance, for a real-valued function ep,
k aep
</l(T n ) - </lee) = 8 aXi (e)(Tn,i - e i )
1 k k a 2 ep
+-"" C e )c T. .-e. )c T. .-e. ) +....
2 a . a . n,l l n,J J
i=l }=1 X l XJ
If the sequences Tn,i - e i are of different order, then it may happen, for instance, that the
linear part involving Tn,i - e i is of the same order as the quadratic part involving CTn,} - e})2.
Thus, it is necessary to determine carefully the rate of all terms in the expansion, and to
rearrange these in decreasing order of magnitude, before neglecting the "remainder."
*3.4 Uniform Delta Method
Sometimes we wish to prove the asymptotic normality of a sequence .Jli(epCT n ) - epce n ))
for centering vectors en changing with n, rather than a fixed vector. If .JliCe n - e) h for
certain vectors e and h, then this can be handled easily by decomposing
.Jli ( ep CTn) - ep C en)) == .Jli (ep CTn) - ep ce)) - .Jli (ep C en) - ep ce) ) .
3.5 Moments
33
Several applications of Slutsky's lemma and the delta method yield as limit in law the vector
ep (T + h) - ep (h) == ep (T), if T is the limit in distribution of (Tn - en). For en --+ e
at a slower rate, this argument does not work. However, the same result is true under a
slightly stronger differentiability assumption on ep.
3.8 Theorem. Let ep : JR.k JRm be a map defined and continuously differentiable in
a neighborhood of e. Let Tn be random vectors taking their values in the domain of
ep. If rn(Tn - en) T for vectors en -+ e and numbers r n -+ 00, then rn(ep(Tn) -
ep (en) ) ep (T) . Moreover, the difference between r n (<P (Tn) - <p (en) ) and ep (r n CTn - en) )
converges to zero in probability.
Proof. It suffices to prove the last assertion. Because convergence in probability to zero
of vectors is equivalent to convergence to zero of the components separately, it is no loss
of generality to assume that ep is real-valued. For O < t < 1 and fixed h, define gn (t) ==
ep (en + t h). For sufficiently large n and sufficiently small h, both en and en + h are in a
ball around e inside the neighborhood on which ep is differentiable. Then gn : [O, 1] f-+ IR is
continuously differentiable with derivative g (t) == <P n+th (h). By the mean-value theorem,
gn (1) - gn (O) == g ( ) for some O < < 1. In other words
Rn(h):== ep (en + h) - <p (en) - <p (h) == ep n+ h(h) - <p (h).
By the continuity of the map e f-+ <p , there exists for every s > O a o > O such that
11<p (h) - ep (h) II < sllh II for every lIš - eli < o and every h. For sufficiently large n and
II hil < 0/2, the vectors en + h are within distance o of e, so that the norm II Rn (h) II of the
right side of the preceding display is bounded by s II hil. Thus, for any 1] > O,
p(rnIIRn(Tn -en)11 > TJ) < P(IITn -en II > ) +P(rnIlTn -en Ile > TJ).
The first term converges to zero as n -+ 00. The second term can be made arbitrarily small
by choosing s small. .
*3.5 Moments
So far we have discussed the stability of convergence in distribution under transformations.
We can pose the same problem regarding moments: Can an expansion for the moments of
ep (Tn) - ep(e) be derived from a similar expansion for the moment s of Tn - e? In principle
the answer is affirmative, but unlike in the distributional case, in which a simple derivative
of ep is enough, global regularity conditions on <p are needed to argue that the remainder
terms are negligible.
One possible approach is to apply the distributional delta method first, thus yielding the
qualitative asymptotic behavior. Next, the convergence of the moments of ep (Tn) - <p (e)
(or a remainder term) is amatter of uniform integrability, in view of Lemma 2.20. If
ep is uniformly Lipschitz, then this uniform integrability folldws from the corresponding
uniform integrability of Tn - e. If <p has an unbounded derivative, then the connection
between moment s of <p CTn) - <p (e) and Tn - e is harder to make, in general.
34
Delta Method
Notes
The Delta method belongs to the folklore of statisties. It is not entirely trivial; proofs are
sometirnes based on the rnean-value theorem and then require continuous differentiability in
a neighborhood. A generalization to functions on intinite-dimensional spaces is discussed
in Chapter 20.
PROBLEMS
1. Find the joint limit distribution of (--Jii( X - fJ.-), --Jii(S2 - (J2)) if X and S2 are based on a sample
of size n from a distribution with finite fourth moment. Under what condition on the under1ying
distribution are --Jii( X - fJ.-) and --Jii(S2 - (J2) asymptotically independent?
2. Find the asymptotic distribution of --Jii (r - p) if r is the correlation coefficient of a sample of n
bivariate vectors with finite fourth moments. (This is quite a bit of work. It helps to assume that
the mean and the variance are equal to O and 1, respectively.)
3. Investigate the asymptotic robustness of the level of the t -test for testing the mean that rejects
Ha : fJ.- < O if --Jii X / S is larger than the upper ex quantile of the t n -l distribution.
4. Find the limit distribution of the sample kurtosis kn == n-I I: 7= 1 (X ž - X )4 / S4 - 3, and design an
asymptotic level ex test for normality based on kn. (Warning: At least 500 observations are needed
to make the normal approximation work in this case.)
5. Design an asymptotic level ex test for normality based on the sample skewness and kurtosis jointly.
6. Let XI, . . . , X n be i.i.d. with expectation fJ.- and variance 1. Find constants such that an ( X - b n )
converges in distribution if fJ.- == O or fJ.- =1= O.
7. Let XI, . . . , X n be arandom sample from the Poisson distribution with mean (). Find a variance
stabilizing transformation for the sample mean, and construct a confidence interval for () based on
this.
8. Let XI, . . . , X n be i.i.d. with expectation 1 and finite variance. Find the limit distribution of
--Jii( X 1 - 1). If the random variables are sampled from a density f that is bounded and strictly
positive in a neighborhood of zero, show that El X I, == 00 for every n. (The density of X n is
bounded away from zero in a neighborhood of zero for every n.)
4
Moment Estimators
The method ofmoments determines estimators by comparing sample and
theoretical moments. Moment estimators are us efu 1 for their simplicity,
aZthough not always optimal. Maximum likelihood es tima to rs for fuU ex-
ponential families are moment estimators, and their asymptotic normality
can be proved by treating them as such.
4.1 Method of Moments
Let XI, . . . , X n be a samp1e from a distribution P() that depends on a parameter e, ranging
over some set 8. The method of moments consists of estimating e by the s01ution of a
system of equations
1 n
- L fj(X i ) == E()fj(X), j == 1, ..., k,
n i=1
for given functions fl, . . . , fk. Thus the parameter is chosen such that the sample moment s
(on the left side) match the theoretical moments. If the parameter is k-dimensional one
usual1y tries to match k moments in this manner. The choices fj (x) == x j lead to the
method of moments in its simplest forme
Moment estimators are not necessari1y the best estimators, but under reasonab1e condi-
tions they have convergence rate vii and are asymptotically norma!. This is a consequence
of the delta method. Write the given functions in the vec tor notation f == (fl, . . . , fk), and
let e : e r+ }Rk be the vector-valued expectation e(e) == P() f. Then the moment estimator
en solves the system of equations
1 n
IfDnf - - L f(X i ) == e(e) = p()f.
n i=1
For existence of the moment estimator, it is necessary that the vector IPn f be in the range
of the function e. If e is one-to-one, then the moment estimator is uniquely determined as
" -1
en == e (IPnf) and
vn (en - e o ) == vn ( e -1 (IP n f) - e -1 (p()o f) ) .
If IPnf is asymptotically norma1 and e- 1 is differentiable, then the right side is asymptoti-
cally normal by the delta method.
35
36
Moment Estimators
The derivative of e- 1 at e(e o ) is the inverse e 1 of the derivative of e at e o . Because the
function e- 1 is often not explicit, it is convenient to ascertain its differentiability from the
differentiability of e. This is possible by the inverse function theorem. According to this
theorem a map that is (continuously) differentiable throughout an open set with nonsingular
derivatives is locally one-to-one, is of full rank, and has a differentiable inverse. Thus we
obtain the following theorem.
4.1 Theorem. Suppose that e(e) == Pe f is one-to-one on an open set e c JRk and con-
tinuously differentiable at e o with nonsingular derivative e . Moreover, assume that
A o
Peo II f 11 2 < 00. Then moment estimators en exist with probability tending to one and
satisfy
r:: A e o ( 1-1 T ( 1-1 ) T )
Y n (en - e o ) N O, ee o Peoff ee o .
Proof. Continuous differentiability at e o presumes differentiability in a neighborhood and
the continuity of e 1--+ e and nonsingularity of e o imply nonsingularity in a neighborhood.
Therefore, by the inverse function theorem there exist open neighborhoods U of e o and
V of Pe o f such that e : U 1--+ V is a differentiable bijection with a differentiable inverse
e- 1 : V 1--+ U. Moment estimators en == e- 1 (Pnf) exist as soon as W>nf E V, which
happens with probability tending to 1 by the law of large numbers.
The central limit theorem guarantees asymptotic normality of the sequence (P n f -
Peo f). N ext use Theorem 3.1 on the display preceding the statement of the theorem. .
For completeness, the following two lemmas constitute, if combined, a pro of of the
inverse function theorem. If necessary the preceding theorem can be strengthened somewhat
by applying the lemmas directly. Furthermore, the first lemma can be easily generalized to
infinite-dimensional parameters, such as used in the semiparametric models discussed in
Chapter 25.
4.2 Lemma. Let 8 C JRk be arbitrary and let e : e 1--+ JRk be one-to-one and differentiable
at a point e o with a nonsingular derivative. Then the inverse e- 1 (defined on the range of
e) is differentiable at e(e o ) provided it is continuous at e(e o ).
Proof. Write 17 == e(eo) and h == e- 1 (17 + h) - e- 1 (17). Because e- 1 is continuous at 17,
we have that h 1--+ O as h 1--+ O. Thus
17 + h == e e- 1 (17 + h) == e( h + e o ) == e(e o ) + e o ( h) + 0(11 h II),
as h 1--+ O, where the last step follows from differentiability of e. The displayed equation
can be rewritten as e o ( h) == h +0(11 h II). By continuity of the inverse of e o' this implies
that
h == e 1 (h) + 0(11 h II).
Inparticular, II hll(l+o(I)) < Ile I(h)11 == O(llhll). Insertthisinthedisplayedequation
to obtain the desired result that h == e 1 (h) + o(llh II). .
4.3 Lemma. Let e : e 1--+ JRk be defined and differentiable in a neighborhood of a point
e o and continuously differentiable at e o with a nonsingular derivative. Then e maps every
4.2 Exponential Families
37
sufficiently small open neighborhood U of80 onto an open set V and e- 1 : V U is well
defined and continuous.
Proof. By assumption, e A- 1 :== e o as 8 8 0 . Thus II I - Ae II < for every
8 in a sufficiently small neighborhood U of 8 0 . Fix an arbitrary point 'TJ1 == e(e 1 ) from
V == e(U) (where 8 1 EU). Next find an 8 > O such that ball(81, 8) eU, and fix an
arbitrary p oint 17 with II 'TJ - 'TJ 1 II < 8 :== II A 11- 1 8. It will be shown that 'TJ e(8) for some
point 8 E ball(8 1 , 8). Renee every 1] E ball('TJ1, 8) has an original in ball(8 1 , 8). If e is
one-ta-one on U, so that the original is unique, then it follows that V is open and that e- 1
is continuous at 'TJ1.
Detine a function ep (8) == 8 + A ('TJ - e (8) ). Because the norm of the derivative ep ==
I - Ae is bounded by throughout U, the map ep is a contraction on U. Furthermore, if
118 - el II < 8,
1
II ep (e) - el II < Ilep(e) - ep(e 1 ) II + II ep (el) - el II < 211e - e 1 11 + IlA 111117 - 17111 < 8.
Consequently, ep maps ball (8 1 , 8) into its elf. Because ep is a contraction, it has a tixed point
8 E ball(e 1 , 8): a point with ep(8) == 8. By detinition of ep this satisties e(e) == 'TJ.
Any other e with e (e) == 'TJ is also a fixed point of ep. In that case the difference e - 8 ==
ep (e) - ep (e) has narm bounded by II e - 8 II. This can only happen if e == e. Renee e is
one-ta-one thraughout U. .
4.4 Example. Let XI, . . . , X n be arandom sample from the beta-distribution: The com-
mon density is equal to
r(a + f3) a-1 f3-1
X x (1 - x) 1 0 <x<1.
r(a)r(,8)
The moment estimator for (a, ,8) is the solution of the system of equations
a
X n == E a RX 1 == ,
,fJ a + ,8
_ 2 2 (a + l)a
X - E f3X -
n - a, 1 - (a + f3 + l)(a + f3) .
The righthand side is a smooth and regular function of (a, f3), and the equations can be
solved explicitly. Renee, the moment estimators exist and are asymptotically norma!. D
*4.2 Exponential Families
Maximum likelihood estimators in full exponential families are moment estimators. This can
be exploited to show their asymptotic normality. Actually, as shown in Chapter 5, maximum
likelihood estimators in smoothly parametrized models are asymptotically normal in great
generality. Therefore the present section is included for the benefit of the simple proof,
rather than as an explanation of the limit properties.
Let XI, . . . , X n be a sample from the k-dimensional exponential family with density
pe (x) == c(8) h (x) ee T t(x).
38
Moment Estimatars
Thus h and t == (tI, . . . , tk) are known functions on the sample space, and the family is
given in its natural parametrization. The parameter set e must be contained in the natural
parameter space for the family. This is the set of e for which Pe can define a probability
density. If f.L is the dominating measure, then this is the right side in
e c {e E ]Rk : c(e)-l = f h(x) i JT t(x) df.L(x) < oo}.
It is a standard result (and not hard to see) that the natural parameter space is convex. It is
usually open, in which case the family is called "regular." In any case, we assume that the
true parameter is an inner point of e. Another standard result concerns the smoothness of
the function e r--+ c (e), or rather of its inverse. (For a proof of the following lemma, see
[100, p. 59] or [17, p. 39].)
4.5 Lemma. Thefunctione r--+ f h(x) eeTt(x) df.L(x) isanalyticontheset{e E Ck: Re e E
o
e }. Its derivatives can be found by differentiating (repeatedly) under the integral sign:
aP f h(x)eeTt(x)df.L(x) f . . T
. . == h(x) tI (XYl . . . tk(XYk ee t(x) df.L(x),
ae l . . . ae{k
for any natural numbers p and il + . . . + ik == p.
The lemma implies that the log likelihood le (x) == log pe (x) can be differentiated (in-
finitely often) with respect to e. The vector of p arti al derivatives (the score function) satisfies
. c
le(x) == -ce) + t(x) == t(x) - Eet(X).
c
Here the second equality is an example of the general rule that score functions have zero
means. It can formally be established by differentiating the identity f Pe d f.L = 1 under the
integral sign: Combine the lemma and the Leibniz rule to see that
f pe df.L == f ac(e) h(x) e()Tt(x) df.L(x) + f c(e) h(x) ti(X) eeTt(x) df.L(x).
ae i ae i
The left side is zero and the equation can be rewritten as O == c / c (e) + Ee t (X).
It follows that the likelihood equations L fe (Xi) == O reduce to the system of k equations
1 n
- L t(X i ) == Eet(X).
n . 1
l=
Thus, the maximum likelihood estimators are moment estimators. Their asymptotic prop-
erties depend on the function e (e) == Ee t (X), which is very well behaved on the interior
of the natural parameter set. By differentiating Eet (X) under the expectation sign (which
is justified by the lemma), we see that its derivative matrices are given by
e == Cove t(X).
The exponential family is said to be offull rank if no linear combination L =1 A jtj (X) is
constant with probability 1; equivalently, if the covariance matrix of t eX) is nonsingular. In
4.2 Exponential Families
39
view of the preceding display, this ensures that the derivative e is strictly positive-definite
throughout the interior of the natural parameter set. Then e is one-to-one, so that there exists
at most one solution to the moment equations. (Cf. Problem 4.6.) In view of the expression
for fe, the matrix -ne is the second-derivative matrix (Hessian) of the log likelihood
L:7=1 le (Xi). Thus, a solution to the moment equations must be a point of maximum of
the log likelihood.
A solution can be shown to exist (within the natural parameter space) with probability
1 if the exponential family is "regular," or more generally "steep" (see [17]); it is then a
point of absolute maximum of the likelihood. If the true parameter is in the interior of the
parameter set, then a (unique) solution en exists with probability tending to 1 as n r--+ 00,
in any case, by Theorem 4.1. Moreover, this theorem shows that the sequence ,Jn (en - e o )
is asymptotically normal with covariance matrix
e o -1 Coveo t (X) (e o -1) T == (Coveo t (X)) -1.
So far we have considered an exponential family in standard form. Many examples arise
in the form
pe (x) == d (e) h(x) eQ(e)T t(x) ,
(4.6)
where Q == (Q1, . . . , Qk) is a vector-valued function. If Q is one-to-one and a maximum
likelihood estimator en exists, then by the invariance of maximum likelihood estimators
under transformations, Q (en) is the maximum likelihood estimator for the natural parameter
Q(e) as considered before. If the range of Q contains an open ball around Q(e o ), then
the preceding discussion shows that the sequence ,Jn (Q (en) - Q (e o )) is asymptotically
norma!. It requires another application of the delta method to obtain the limit distribution
of ,Jn (en - e o ). As is typical of maximum likelihood estimators, the asymptotic covariance
matrix is the inverse of the Fisher information matrix
. . T
le == Eele (X)le (X) .
4.6 Theorem. Let 8 C }Rk be open and let Q : e r--+ }Rk be one-to-one and continuously
differentiable throughout e with nonsingular derivatives. Let the (exponential) family of
densities pe be given by (4.6) and be offuli ranko Then the likelihood equations have a
unique solution en withprobability tending to 1 and ,Jn(e n - e) N(G, le- 1 )forevery e.
Proof. According to the inverse function theorem, the range of Q is open and the inverse
map Q-1 is differentiable throughout this range. Thus, as discussed previously, the delta
method ensures the asymptotic normality. It suffices to calculate the asymptotic covariance
matrix. By the preceding discussion this is equal to
Q -l (Cove t (X))-l (Q -l)T.
. .
By direct calculation, the score function for the model is equal to le (x) d / d (e) +
(Q )T t(x). As before, the score function has mean zero, so that this can be rewritten
as le(x) == (Q )T(t(x) - Eet(X)). Thus, the Fisher information matrix equals le ==
(Q ) T Cove t (X) Q . This is the inverse of the asymptotic covariance matrix given in the
preceding display. .
40
Moment Estimatars
Not all exponential families satisfy the conditions of the theorem. For instance, the
normal N (e, ( 2 ) family is an example of a "curved exponential family." The map Q (e) ==
(e- 2 , e-l) (with t(x) == (-x 2 j2, x)) does not fiH up the natural parameter space of the
normallocation-scale family but only traces out a one-dimensional curve. In such cases the
result of the theorem may stiH hold. In fact, the result is true for most models with "smooth
parametrizations," as is seen in Chapter 5. However, the "easy" proof of this section is not
valid.
PROBLEMS
1. Let XI, . . . , X n b e a sample from the uniform distribution on [ -f), f)]. Find the moment estimator
of f) based on X2. Is it asymptotically normal? Can you think of an estimator for f) that converges
faster to the parameter?
2. Let XI, . . . , X n be a sample from a density pe and f a function such that e(f)) == Ee f (X) is
differentiable with e' (f)) == Eefe (X) f (X) for le == log pe.
(i) Show that the asymptotic variance of the moment estimator based on f equals vare (f) /
COVe (f, fe)2.
(ii) Show that this is bigger than Ie- l with equality for all f) if and only if the moment estimator
is the maximum likelihood estimator.
(iii) Show that the latter happens only for exponential family members.
3. To what extent does the result of Theorem 4.1 require that the observations are i.i.d.?
4. Let the observations be a sample of size n from the N(J1-, 0- 2 ) distribution. Calculate the Fisher
information matrix for the parameter f) == (J1-, o- 2 ) and its inverse. Check directly that the maximum
likelihood estimator is asymptotically normal with zero mean and covariance matrix Ii 1 .
5. Establish the formula e == Cove t (X) by differentiating e(f)) == Eet (X) under the integral sign.
(Differentiating under the integral sign is justified by Lemma 4.5, because Ee t (X) is the first
derivative of c (f)) -1 .)
6. Suppose a function e: 8 r-+ k is defined and continuously differentiable on a convex subset
8 c k with strictly positive-definite derivative matrix. Then e has at most one zero in 8.
(Consider the function geA) == (f)l - f)2)T e(Af)l + (1 - A)f)2) for given f)l i= f)2 and O < A < 1.
If g(O) == g(l) == O, then there exists a point AO with g' (Ao) == O by the mean-value theorem.)
5
M- and Z-Estimators
This chapter gives an introduction to the consistency and asymptotic
normality of M -estimators and Z -estimators. Maximum likelihood esti-
mators are treated as a special case.
5.1 Introduction
Suppase that we are interested in a parameter (or "functional") f) attached to the distribution
ofobservationsX 1 ,..., Xn. Apopularmethodforfindinganestimatore n ==e n (X 1 ,..., X n )
is to maximize a criterion function of the type
1 n
f) r-+ Mn(f)) == - Lme(Xi).
n i=l
(5.1)
Here me: X r-+ IR are known functions. An estimator maximizing M n (f)) over e is called
an M -estimator. In this chapter we investigate the asymptotic behavior of sequences of
M -estimators.
Often the maximizing value is sought by setting a derivative (or the set of p arti al deriva-
tives in the multidimensional case) equal to zero. Therefore, the name M -estimator is also
used for estimators satisfying systems of equations of the type
1 n
\lin (f)) == - L 1/1e (Xi) == O.
n. 1
l=
(5.2)
Here 1/1e are known vector-valued maps. For instance, if f) is k-dimensional, then 1/1e
typically has k coordinate functions l/1e == (l/1e, 1, . . . , l/1e,k), and (5.2) is shorthand for the
system of equations
n
L l/1e,j (Xi) == O,
i=l
j == 1, 2, .. . , k.
Even though in many examples l/1e,j is the jth p arti al derivative of some function me, this
is irrelevant for the following. Equations, such as (5.2), defining an estimator are called
estimating equations and need not correspond to a maximization problem. In the latter case
it is probably better to call the corresponding estimators Z -estimators (for zero), but the
use of the name M -estimator is widespread.
41
42
M- and Z-Estimators
Sometimes the maximum of the criterion function M n is not taken or the estimating
equation does not have an exact solution. Then it is natural to use as estimator a value
that almost maximizes the criterion function or is a near zero. This yields approximate
M -estimators or Z-estimators. Estimators that are sufficiently close to being a point of
maximum or a zero often have the same asymptotic behavior.
An operator notation for taking expectations simplifies the formulas in this chapter.
We write P for the marginallaw of the observations XI, . . . , X n , which we assume to be
identically distributed. Furthermore, we write P f for the expectation Ef (X) == f f d P and
abbreviate the average n-I 'L7 = 1 f (X ž) to rp n f. Thus rp n is the empirical distribution: the
(random) discrete distribution that puts mass 1/ n at every of the observations XI, . . . , Xn.
The criterion functions now take the forms
M n (e) == rp nme, and \lin (e) == rp n 1/161.
We also abbreviate the centered sums n- I / 2 'L7=I (f(X ž ) - Pf) to Crn!, the empirical
process at f.
5.3 Example (Maximum likelihood estimators). Suppose XI, ..., X n have a common
density Pe. Then the maximum likelihood estimator maximizes the likelihood n7=I pe (Xi),
or equivalently the log likelihood
n
e f--+ L log Pe (X ž ).
i=I
Thus, a maximum likelihood estimator is an M -estimator as in (5.1) with me == log Pe. If
the density is partially differentiable with respect to e for each fixed x, then the maximum
likelihood estimator also solves an equation of type (5.2), with 1/161 equal to the vec tor of
. .
p arti al derivatives fe,j == B / Be j log Pe. The vector-valued function fe is known as the score
function of the model.
The detinition (5.1) of an M -estimator may apply in cases where (5.2) does not. For
instance, if XI, . . . , X n are i.i.d. according to the uniform distribution on [O, eJ, then it
makes sense to maximize the log likelihood
n
e f--+ L(log I[Q,e](Xd -loge).
i=I
(Define log O == -00.) However, this function is not smooth in e and there exists no natural
version of (5.2). Thus, in this example the definition as the location of a maximum is more
fundamental than the definition as a zero. D
5.4 Example (Location estimators). Let XI, . . . , X n be arandom sample of real-valued
observations and suppose we want to estimate the location of their distribution. "Location"
is a vague term; it could be made precise by defining it as the mean or median, or the center
of symmetry of the distribution if this happens to be symmetric. Two examples of location
estimators are the sample mean and the sample median. Both are Z-estimators, because
they solve the equations
n
L(X ž - e) == O;
ž=I
n
and L sign(X ž - e) == O,
ž=I
5.1 Introduction
43
respectively.t Both estimating equations involve functions of the form 1f;(x - fJ) for a
function 1/1 that is monotone and odd around zero. It seems reasonable to study estimators
that solve a general equation of the type
n
L 1f;(X i - fJ) == O.
i=l
We can consider a Z -estimator defined by this equation a "Iocation" estimator, because it
has the desirable property of location equivariance. If the observations Xi are shifted by a
fixed amount a, then so is the estimate: e + a solves L:7=11f;(X i + a - fJ) == O if e solves
the original equation.
Popular examples are the Huber estimators corresponding to the functions
I -k if x < -k
1fJ(x) = [X] k: = x if IX I < k:
k if x > k.
The Huber estimators were motivated by studies in robust statistics conceming the influ-
ence of extreme data points on the estimate. The exact values of the largest and smallest
observations have very little influence on the value of the median, but a proportional in:flu-
ence on the mean. Therefore, the sample mean is considered nonrobust against outliers. If
the extreme observations are thought to be rather unreliable, it is certainly an advantage to
limit their in:fluence on the estimate, but the median may be too successful in this respect.
Depending on the va1ue of k, the Huber estimators behave more like the mean (1arge k) or
more like the median (small k) and thus bridge the gap between the nonrobust mean and
very robust median.
Another example are the quantiles. A pth sample quantile is roughly a point fJ such that
pn observations are less than fJ and (1 - p)n observations are greater than fJ. The precise
detinition has to take into account that the value pn may not be an integer. One possibility
is to call a pth sample quantile any e that solves the inequalities
n
-1 < L((l - p)l{X i < fJ} - pl {Xi > fJ}) < 1.
i=l
(5.5)
This is an approximate M -estimator for 1f; (x) == 1 - p, O, - p if x < O, x == O, or x > O,
respectively. The "approximate" refers to the inequalities: It is required that the value of
the estimating equation be inside the interval (-1, 1), rather than exactly zero. This may
seem a rather wide tolerance interval for a zero. However, all solutions tum out to have the
same asymptotic behavior. In any case, except for special combinations of p and n, there is
no hope of finding an exact zero, because the criterion function is discontinuous with jumps
at the observations. (See Figure 5.1.) If no observations are tied, then all jumps are of si ze
one and at least one solution e to the inequalities exists. If tied observations are present, it
may be necessary to increase the interval (-1, 1) to ensure the existence of solutions. Note
that the present 1f; function is monotone, as in the previous examples, but not symmetric
about zero (for p =I- 1/2).
t The sign-function is defined as sign(x) = -1, O, 1 if x < O, x = O or x > O, respeetively. Also x+ means
x vO = max(x, O). For the median we as sume that there are no tied observations (in the middle).
44 M - and Z-Estimators
I.!)
o
C\I
o
o T"""
o
Ll(
o
T"""
I
o o
C\I
T""" I
I
4 6 8 10 12 -2 -1 O 2 3
Figure 5.1. The functions e H- \II n (e) for the 80% quantile and the Huber estimator for samples of
size 15 from the gamma(8,1) and standard normal distribution, respectively.
All the estimators considered so far can also be defined as a solution of a maximization
problem. Mean, median, Huber estimators, and quantiles minimize 'L7=1 m(X i - e) for m
equal to x 2 , Ixl, x21lxl:::k + (2klxl - k 2 )1 Ixl >k and (1 - p)x- + px+, respectively. O
5.2 Consistency
If the estimator en is used to estimate the parameter e, then it is certainly desirable that
the sequence 8 n converges in probability to e. If this is the case for every possible value
of the parameter, then the sequence of estimators is called asymptotically consistent. For
instance, the sample mean X n is asymptotically consistent for the population mean EX
(provided the population mean exists). This follows from the law of large numbers. Not
surprisingly this extends to many other sample characteristics. For instance, the sam-
ple median is consistent for the population median, whenever this is well defined. What
can be said about M -estimators in general? We shalI assume that the set of possible
parameters is a metric space, and write d for the metric. Then we wish to prove that
d(8n, ( 0 ) O for some value ea, which depends on the underlying distribution of the
observations.
Suppose that the M -estimator 8 n maximizes the random criterion function
e Mn(e).
Clearly, the "asymptotic value" of 8n depends on the asymptotic behavior of the functions
M n . Under suitable normalization there typically exists a deterministic "asymptotic criterion
function" e M(e) such that
p
M n (e)-+ M(e),
every e.
(5.6)
For instance, if Mn(e) is an average of the form JPnme as in (5.1), then the law of large
numbers gives this result with M(e) == Pme, provided this expectation exists.
It seems reasonable to expect that the maximizer en of M n converges to the maximizing
value e a of M. This is what we wish to prove in this section, and we say that 8 11 is
(asymptotically) consistent for e a . However, the convergence (5.6) is too weak to ensure
5.2 Consistency
45
Figure 5.2. Example of a function whose point of maximum is not well separated.
the convergence of en. Because the value en depends on the whole function e r-+ M n (e),
an appropriate form of "functional convergence" of M n to M is needed, strengthening the
pointwise convergence (5.6). There are several possibilities. In this section we first discuss
an approach based on uniform convergence of the criterion functions. Admittedly, the
assumption of uniform convergence is too strong for some applications and it is sometimes
not easy to verify, but the approach illustrates the general idea.
Given an arbitrary random function e r-+ M n (e), consider estimators en that nearly
maximize M n , that is,
Mn(e n ) > sup Mn(e) - opel).
e
Then certainly Mn(e n ) > Mn(e a ) - opel), which turns out to be enough to ensure con-
sistency. It is assumed that the sequence M n converges to a nonrandom map M: 8 r-+ JR.
Condition (5.8) of the following theorem requires that this map attains its maximum at a
- unique point e a , and only parameters close to e a may yield a value of M (e) close to the
maximum value M(e a ). Thus, (Ja should be a well-separated point of maximum of M.
Figure 5.2 shows a function that does not satisfy this requirement.
5.7 Theorem. Let M n be randomfunctions and let M be afixedfunction ofe such that
for every £ > ot
supIMn(e) - M(e)1 O,
eE8
sup M(e) < M(e a ).
e : d(e,eo) £
(5.8)
Then any sequence of estimators en with Mn(e n ) > Mn(e a ) - opel) converges in proba-
bility to (Ja.
t Some of the expressions in this display may be nonmeasurable. Then the probability statements are understood
in terms of outer measure.
46
M - and Z - Estimators
Proof. By the property of en, we have Mn(e n ) > Mn(e o ) - op(l). Because the uniform
convergence of M n to M implies the convergence of M n (e o ) M(B o ), the right side equals
M (e o ) - o p (1). It follows that M n (en) > M (B o ) - o p (1), whence
M(e o ) - M(e n ) < Mn(e n ) - M(e n ) + op(l)
p
< sup IM n - MI(B) + op(l) --+ O.
e
by the first part of assumption (5.8). By the second part of assumption (5.8), there exists for
every £ > O a number 'YJ > O such that M(B) < M(e o ) - 'YJ for every B with dee, eo) > £.
Thus, the event {d(e n , Bo) > £} is contained in the event {M(e n ) < M(e o ) - 'YJ}. The
probability of the latter event converges to O, in view of the preceding display. .
Instead of through maximization, an M -estimator may be defined as a zero of a criterion
function B r--+ W n (B). It is again reasonable to assume that the sequence of criterion
functions converges to a fixed limit:
p
Wn(e) --+ WeB).
Then it may be expected that a sequence of (approximate) zeros of w n converges in prob-
ability to a zero of W. This is true under similar restrictions as in the case of maximizing
M estimators. In fact, this can be deduced from the preceding theorem by noting that a
zero of \I1n maximizes the function e r--+ -II W n (e) II.
5.9 Theorem. Let \I1n be random vector-valued functions and let W be a fixed vector-
valued function of e such that for every £ > O
sUPeE8 II W n (e) - W (B) II O,
infe: d(e,eo) e II W (B) II > O == II W (e o ) II.
Then any sequence of estimators en such that \li n (e n) == O P (1) converges in probability
to eo.
Proof. This follows from the preceding theorem, on applying it to the functions M n (e) ==
-II W n (e) II and M (e) == -II W (B) II. .
The conditions of both theorems consist of a stochastic and a deterministic part. The
deterministic condition can be verified by drawing a picture of the graph of the function. A
helpful general observation is that, for a compact set e and continuous function M or W,
uniqueness of e o as a maximizer or zero implies the condition. (See Problem 5.27.)
For Mn(e) or Wn(B) equal to averages as in (5.1) or (5.2) the uniform convergence
required by the stochastic condition is equivalent to the set of functions {me: B E e}
or {o/e,j: B E e, j == 1,..., k} being Glivenko-Cantelli. Glivenko-Cantelli classes of
functions are discussed in Chapter 19. One simple set of sufficient conditions is that e be
compact, that the functions e r--+ me (x) or e r--+ o/e (x) are continuous for every x, and that
they are dominated by an integrable function.
Uniform convergence of the criterion functions as in the preceding theorems is much
stronger than needed for consistency. The following lemma is one of the many possibilities
to replace the uniformity by other assumptions.
5.2 Consistency
47
5.10 Lemma. Let 8 be a subset of the real line and let '¥n be random functions and
'¥ a fixed function of e such that '¥ n (e) '¥ (e) in probability for every e. Assume that
each map e f---+ \lin ce) is continuous and has exactly one zero en, or is nondecreasing with
'¥n (en) == o p (1). Let e o be a point such that '¥ (e o - E') < O < '¥ (e o + E') for every E' > O.
" p
Then en eo.
Proof. If the map e r-+ '¥ n (e) is continuous and has a unique zero at en, then
p ( '¥ n ((30 - E') < O, '¥ n (e o + E') > O) < p (e o - E' < en < e o + E').
The left side converges to one, because '¥n (e o :f: E') '¥ (e o :f: E') in probability. Thus the
right side converges to one as well, and en is consistent.
If the map e \II n (e) is nondecreasing and en is a zero, then the same argument is valid.
More generally, if e f---+ \lin (e) is nondecreasing, then '¥n (e o - E') < -17 and en < e o - E'
imply '¥n (en) < -17, which has probability tending to zero for every 17 > O if en is a near
zero. This and a similar argument applied to the right tail shows that, for every E', rJ > O,
p ( '¥ n (e o - E') < -17, '¥ n (e o + E') > 17) < p (e o - E' < en < e o + E') + 0(1).
For 21] equal to the smallest of the numbers - '¥ (e o - E') and '¥ (e o + E') the left side still
converges to one. .
5.11 Example (Median). The sample median en is a(near) zeroofthemape '¥n(e) ==
n- I 2::7=1 sign(X i - e). By the law of large numbers,
'¥nce) '¥(e) == Esign(X - e) == P(X > e) - P(X < e),
for every fixed e. Thus, we expect that the sample median converges in probability to a
point (30 such that P(X > eo) == P(X < e o ): a population median.
This can be proved rigorously by applying Theorem 5.7 or 5.9. However, even though
the conditions of the theorems are satisfied, they are not entirely trivial to verify. (The
uniform convergence of '¥n to '¥ is proved essentially in Theorem 19.1) In this case it
is easier to apply Lemma 5.10. Because the functions e '¥n(e) are nonincreasing, it
follows that e n (30 provided that '¥ (e o - E') > O > '¥ (e o + E') for every E' > O. This is
the case if the population median is unique: P(X < e o - E') < < P(X < e o + E') for all
E' > O. D
*5.2.1 Wald's Consistency Proof
Consider the situation that, for arandom sample of variables XI, . . . , X n,
1 n
Mn(e) == IFnme == - Lme(X i ),
n. 1
1=
M(e) == Pme.
In this subsection we consider an alternative set of conditions under which the maximizer en
of the process M n converges in probability to a point of maximum e o of the function M. This
"classicaI" approach to consistency was taken by Wald in 1949 for maximum likelihood
estimators. It works best if the parameter set 8 is compact. If not, then the argument must
48
M- and Z-Estimators
be complemented by a proof that the estimators are in a compact set eventually or be applied
to a suitable compactification of the parameter set.
Assume that the map () r-+ me (x) is upper-semicontinuous for almost all x: For every ()
limsupmen (x) < me(x),
en -+e
a.s..
(5.12)
(The exceptional set of x may depend on () .) Furthermore, assume that for every sufficiently
small ball U C 8 the function x r-+ sUPeEu me (x) is measurable and satisfies
p sup me < 00.
eEU
(5.13)
Typically, the map () r-+ Pme has a unique global maximum at a point ()o, but we shalI
allow multiple points of maximum, and write 8 0 for the set {80 E 8: Pmeo == sUPe Pme}
of all points at which Mattains its global maximum. The set 8 0 is assumed not empty. The
maps me: X r-+ JR. are allowed to take the value - 00, but the following theorem assumes
implicit1y that at least P meo is fini te.
5.14 Theorem. Let() r-+ me (x) be upper-semicontinuousfor almost all x and let (5.13) be
satisfied. Thenforanyestimatorse n such that Mn(e n ) > M n (()o)-op(l)forsome()o E 8 0 ,
for every £ > O and every compact set K C 8,
p(d(e n , 8 0 ) > £ /\ en E K) --+ O.
Proof. If the function () r-+ Pme is identically -00, then 8 0 == 8, and there is nothing
to proveo Renee, we may assume that there exists 8 0 E 8 0 such that Pmeo > -00, whence
p Ime o I < 00 by (5.13).
Fix some () and let Uz -i () be a decreasing sequence of open balls around () of diameter
converging to zero. Write mu(x) for sUPeEume(x). The sequence muz is decreasing
and greater than me for every l. Combination with (5.12) yields that muz -i me almost
surely. In view of (5.13), we can apply the monotone convergence theorem and obtain that
Pmuz -i Pme (which may be -(0).
For () tt 8 0 , we have Pme < Pmeo. Combine this with the preceding paragraph to see
that for every () tt 8 0 there exists an open ball Ue around () with P m Ue < P meo. The set
B == {() E K: d ((), 8 0 ) > £} is compact and is covered by the balls {Ue: () E B} . Let
Ue! ' . . . , Ue p be a finite subcover. Then, by the law of large numbers,
as
supJPnme < sup JPnmUe. --+ sup pmu e . < Pmeo.
eB 0 1 }. }
E J= ,...,p J
If enE B, then sUPeEB JP nme is at least JP nmen, which by definition of en is at least JP nmeo -
o p (1) == Pmeo - o p (1), by the law of large numbers. Thus
{ enE B} C { SUP JP n m e > P meo - O P ( 1) } .
eEB
In view of the preceding display the probability of the event on the right side converges to
zero as n --+ 00. .
Even in simple examples, condition (5.13) can be restrictive. One possibility for relax-
ation is to divide the n observations in groups of approximately the same size. Then (5.13)
5.2 Consistency
49
may be replaced by, for some k and every k < l < 2k,
I
pl sup Lme(xi) < 00.
eEU i=l
(5.15)
Surprisingly enough, this simple device may help. For instance, under condition (5.13)
the preceding theorem does not apply to yield the asymptotic consistency of the maximum
likelihood estimator of (IL, 0-) based on arandom sample from the N(IL, 0- 2 ) distribution
(unless we restrict the parameter set for 0-), but under the relaxed condition it does (with
k == 2). (See Problem 5.25.) The pro of of the theorem under (5.15) remains almost the
same. Divide the n observations in groups of k observations and, possibly, a remainder
group of l observations; next, apply the law of large numbers to the approximately n / k
group sums.
5.16 Example (Cauchy likelihood). The maximum likelihood estimator for e based on a
random sample from the Cauchy distribution with location e maximizes the map e 1---+ IfD nme
for
me(x) == -log(l + (x - e)2).
The natural parameter set JR is not compact, but we can enlarge it to the extended real line,
provided that we can define me in a reasonable way for e == ::f:oo. To have the best chance
of satisfying (5.13), we opt for the minimal extension, which in order to satisfy (5.12) is
m -00 (x) == lim sup me (x) == - 00;
e t---+ - 00
moo(x) == lim sup me(x) == -00.
e t---+ 00
These infinite values should not worry us: They are permitted in the preceding theorem.
Moreover, because we maximize e 1---+ IfD nme, they ensure that the estimator en never takes
the values ::f:oo, which is excellent.
We apply Wald' s theorem with 8 == IR, equipped with, for instance, themetric d(e l , e 2 ) ==
I arctg el - arctg e 2 1. Because the functions e 1---+ me (x) are continuous and nonpositive, the
conditions are trivially satisfied. Thus, taking K == IR , we obtain that d (en, ( 0 ) O. This
conclusion is valid for any underlying distribution P of the observations for which the set
8 0 is nonempty, because so far we have used the Cauchy likelihood only to motivate me.
To conclude that the maximum likelihood estimator in a Cauchy location model is con-
sistent, it suffices to show that 80 == {e o } if P is the Cauchy distribution with center e o . This
follows most easily from the identifiability of this model, as discussed in Lemma 5.35. D
5.17 Example (Current status data). Suppose that a "death" that occurs at time T is only
observed to have taken place or not at a known "check-up time" C. We model the obser-
vations as arandom sample XI, . . . , X n from the distribution of X == (C, 1 {T < C}),
where T and C are independent random variables with completely unknown distribution
functions F and G, respectively. The purpose is to estimate the "survival distribution"
1 - F.
If G has a density g with respect to Lebesgue measure A, then X == (C, ) has a density
PF(C, 8) == (8F(c) + (1 - 8)(1 - F)(c))g(c)
50
M- and Z-Estimators
with respect to the product of A and counting measure on the set {O, I}. A maximum like-
lihood estimator for F can be defined as the distribution function P that maximizes the
likelihood
n
F r-+ n( iF(Ci) + (1 - i)(l- F)(C i ))
i=l
over all distribution functions on [O, (0). Because this only involves the numbers F(C 1 ),
. . . , F (C n), the maximizer of this expression is not unique, but some thought shows that
there is a unique maximizer P that concentrates on (a subset of) the observation times
Cl, . . . , C n. This is commonl y used as an estimator.
We can show the consistency of this estimator by Wald's theorem. By its definition P
maximizes the function F r-+ IP n log p p, but the consistency proof proceeds in a smoother
way by setting
mp == log
p(P +Po)j2
PP
2p p
== log .
P p + p Po
Because the likelihood is bigger at P than jt is at P + Fo, it follows that IP n m ft > O ==
IPnmpo' (It is not claimed that P maximizes F r-+ IPnmp; this is not true.)
Condition (5.13) is satisfied trivially, because m P < log 2 for every F. We can equip the
set of all distribution functions with the topology of weak convergence. If we restrict the
parameter set to distributions on a compact interval [O, T], then the parameter set is compact
by Prohorov' s theorem. t The map F r-+ m P (c, 8) is continuous at F, relative to the weak
topology, for every (e, 8) such that c is a continuity point of F. Under the assumption that
G has a density, this includes almost every (e, 8), for every given F. Thus, Theorem 5.14
shows that P n converges under Fo in probability to the set :Fo of all distribution functions
that maximize the map F r-+ Ppomp, provided Fo E :Fo. This set always contains Fo, but
it does not necessarily reduce to this single point. For instance, if the density g is zero
on an interval [a, b], then we receive no information concerning deaths inside the interval
[a, b], and there can be no hope that P n converges to Fo on [a, b]. In that case, Fo is not
"identifiable" on the interval [a, b].
We shalI show that :Fo is the set of all F such that F == Fo almost everywhere according
to G. Thus, the sequence P n is consistent for Fo "on the set of time points that have a
positive probability of occurring."
Because p P == P Po under PPo if and only if F == Fo almost everywhere according to G, it
suffices to prove that, for every pair of probability densities p and Po, Po log 2 P j (p+ Po) < O
with equality if and only if P == po almost surely under Po. If Po (p == O) > O, then
log 2p j (p + Po) == -00 with positive probability and hence, because the function is
bounded above, P o log2pj(p + Po) == -00. Thus we may assume that Po(p == O) == O.
Then, with f(u) == -u log( + u),
2p ( Po ) ( Po )
Polog ==Pf - < f P- ==f(l) ==0,
(p + Po) P P
t Altematively, eonsider all probability distributions on the eompaetifieation [O, 00] again equipped with the
weak topology.
5.3 Asymptotic Normality
51
by Jensen's inequality and the concavity of f, with equality only if Pol P == 1 almost surely
under P, and then also under Po. This completes the pro of. O
5.3 Asymptotic Normality
Suppose a sequence of estimators en is consistent for a parameter () that ranges over an
open subset of a Euclidean space. The next question of interest concerns the order at which
the discrepancy en - g converges to zero. The answer depends on the specific situation,
but for estimators based on n replications of an experiment the order is often n- 1j2 . Then
multiplication with the inverse of this rate creates a proper balance, and the sequence
,Jn (en - ()) converges in distribution, most often a normal distribution. This is interesting
from a theoretical point of view. It also makes it possible to obtain approximate confidence
sets. In this section we derive the asymptotic normality of M -estimators.
We can use a characterization of M -estimators either by maximization or by solving
estimating equations. Consider the second possibility. Let XI, . . . , X n be a sample from
some distribution P, and let arandom and a "true" criterion function be of the form:
1 n
\lin «()) = - L lfre (Xi) == JP> n lfre,
n. 1
l=
\lI(g) == Plfre.
Assume that the estimator Đ n is a zero of \lin and converges in probability to a zero ()a of \li.
Because en --+ ()a, it makes sense to expand \lin (en) in a Taylor series around ()a. Assume
for simplicity that e is one-dimensiona1. Then
AA' 1 A 2"-
O == \II n «() n) == \li n (gO) + «() n - e O ) \II n (gO) + "2 ( g n - gO) \li n (en) ,
where en is a point between Đ n and e a . This can be rewritten as
C A -,Jn\lln (e o )
v n ( g n - go) == . 1 A .. - .
\lin (gO) + "2 (gn - e O ) \lin (g n)
If Plfr is finite, then the numerator -,Jn\lln(e o ) == _n- 1j2 L lfreo(X i ) is asymptotically
normal by the central limit theorem. The asymptotic mean and variance are P lfreo ==
\li (e a ) == O and Plfr o' respectively. Next consider the denominator. The first term W n (go)
. p.
is an average and can be analyzed by the law of large numbers: \lin (go) --+ Plfr eo' provided
the expectation exists. The second term in the denominator is a product of en - e == O P (1)
and 1/1 n (en) and converges in probability to zero under the reasonable condition that W n (en)
(which is also an average) is O p (1). Together with Slutsky's lemma, these observations
yield
(5.18)
C A ( Plfr o )
v n (g n - e a ) N O, . 2 .
( p lfr e o )
(5.19)
The preceding derivation can be made rigorous by imposing appropriate conditions, often
called "regularity conditions." The only real challenge is to show that W n (en) == O p (1)
(see Problem 5.20 or section 5.6).
The derivati on can be extended to higher-dimensional parameters. For a k-dimensional
parameter, we use k estimating equations. Then the criterion functions are maps \lin :}Rk r+
52
M - and Z-Estimators
JRk and the derivatives n (e o ) are (k x k)-matrices that converge to the (k x k) matrix P e o
with entries pa / ae j 1/I e o,i. The final statement becomes
-Ji!(e n - ( 0 ) Nk (O, (PVr eJ -1 P1/1 e o 1/1!O (PVr J -1) .
(5.20)
Here the invertibility of the matrix P 1/1 e o is a condition.
In the preceding derivation it is implicitly understood that the function e 1---+ 1/Ie (x)
possesses two continuous derivatives with respect to the parameter, for every x. This is true
in many examples but fails, for instance, for the function 1/Ie (x) == sign(x - e), which yields
the median. N evertheless, the median is asymptotically norma!. That such a simple, but
important, example cannot be treated by the preceding approach has motivated much effort
to derive the asymptotic normality of M -estimators by more refined methods. One result
is the following theorem, which assumes less than one derivative (a Lipschitz condition)
instead of two derivatives.
5.21 Theorem. For each e in an open subset of Euclidean space, let x f--+ 1/Ie (x) be a mea-
surable vector-valued function such that, for every el and e 2 in a neighborhood of e o and
. . 2
a measurable function 1/1 with P 1/1 < 00,
111/Ie 1 (x) - 1/Ie 2 (x) II < (x) IIe 1 - e 2 11.
Assume that P 111/I e o 11 2 < 00 and that the map e f--+ p 1/Ie is differentiable at a zero e o , with
nonsingular derivative matrix Veo' IfTI?n1/l 8 n == op(n- 1j2 ), and en e o , then
r:: A -1 1
'\I n (en - e o ) == - Veo 1/Ieo (Xi) + o p (1),
n. 1
1=
In particular, the sequence (en - e o ) is asymptotically normal with mean zero and
. . Y -1 P '1 Ir '1lrT ( y -1 ) T
covarlance matrlx e o 'f' e o 'f' eo eo .
Proof. For a fixed measurable function f, we abbreviate (TIDn - P)f to CGnf, the
empirical process evaluated at f. The consistency of en and the Lipschitz condition on the
maps e f--+ 1/Ie imply that
p
CG n 1/I 8 n - Crn 1/Ie o --+ O.
(5.22)
For a nonrandom sequence en this is immediate from the fact that the means of these variables
are zero, while the variances are bounded by P 111/Ie n - 1/Ie o 11 2 < P 2 11 en - e o 11 2 and hence
converge to zero. A proof for estimators en under the present mild conditions takes more
effort. The appropriate tools are developed in Chapter 19. In Example 19.7 it is seen that
the functions 1/Ie form a Donsker class. Next, (5.22) follows from Lemma 19.24. Here we
accept the convergence as a fact and give the remainder of the proof.
By the definitions of en and e o , we can rewrite Cr n 1/l8/1 as P(1/Ieo - 1/18/1) + op(l).
Combining this with the delta method (or Lemma 2.12) and the differentiability of the map
e 1---+ p 1/Ie, we find that
-Ji! Veo (e o - en) + -Ji! o p (lien - e o II) == G n lf;e o + o p (1).
5.3 Asymptotic Normality
53
In particular, by the invertibility of the matrix Veo,
,Jn!le n - e o II < "Ve lll ,Jnll Veo (en - e o ) II == O p (1) + o p (,Jnlle n - e o II).
This implies that en is ,Jn-consistent: The left side is bounded in probability. Inserting this
in the previous display, we obtain that ,JnV eo (en -e o ) == -((;n 1/1 e o +op(l). We conclude the
proof by taking the inverse Ve 1 left and right. Because matrix mu1tiplication is a continous
map, the inverse of the remain der term stiH converges to zero in probability. .
The preceding theorem is a reasonable compromise between simplicity and general
applicability, but, unfortunately, it does not cover the sample median. Because the function
e sign(x - e) is not Lipschitz, the Lipschitz condition is apparently stiH stronger
than necessary. Inspection of the proof show s that it is used only to ensure (5.22). It is
seen in Lemma 19.24, that (5.22) can be ascertained under the weaker conditions that the
collection of functions x 1/1e (x) are a "Donsker class" and that the map e 1/1e is
continuous in probability. The functions sign(x - e) do satisfy these conditions, but a proof
and the detinition of a Donsker class are deferred to Chapter 19.
If the functions e 1/1e (x) are continuously differentiable, then the natural candidate
. .
for 1/1 (x) is sUPe 111/1 eli, with the supremum taken over a neighborhood of e o . Then the
main condition is that the p arti al derivatives are "locaHy dominated" by a square- integrable
. . .
function: There should exist a square-integrable function 1/1 with 111/1 eli < 1/1 for every e
close to e o . If e 1/1 e (x) is also continuous at e o , then the dominated-convergence theorem
readil y yields that Veo == p 1/1 eD .
The properties of M estimators can typicaHy be obtained under milder conditions by
using their characterization as maximizers. The following theorem is in the same spirit
as the preceding one but does cover the median. It concerns M -estimators detined as
maximizers of a criterion function e JID nme, which are assumed to be consistent for a
point of maximum e o of the function e P me. If the latter function is twice continuously
differentiable at e o , then, of course, it allows a two-term Taylor expansion of the form
Pme == Pmeo + (e - eO)TVeo(e - e o ) + o(lle - 8 0 11 2 ).
It is this expansion rather than the differentiability that is needed in the foHowing theorem.
5.23 Theorem. For each 8 in an open subset of Euclidean space let x me (x) be a mea-
surable function such that e me (x) is differentiable at 8 0 for P -almost every x t with
derivative meo (x) and such that, for every el and e 2 in a neighborhood of e o and a measur-
ablefunction m with Prh 2 < 00
Ime l (x) - me 2 (x) I < m(x) IIe 1 - e 2 11.
Furthermore, assume that the map 8 Pme admits a second-order Taylor expansion
at a point of maximum e o with nonsingular symmetric second derivative matrix Veo' If
1 "p
JIDnm en > SUPe ItD nme - o p (n - ) and 8 n ---+ e o , then
r:: " -1 1 .
v n (8n - 8 0 ) == -Veo ,Jn meo(Xi) + op(l).
n. 1
1=
t Alternatively, it suffices that e me is differentiable at eo in P -probability.
54
M - and Z -Estimators
In particular, the sequence -Jfi(e n - €lo) is asymptotically normal with mean zero and
covariance matrix Ve l Pmeom Ve l.
* Pro of. The Lipschitz property and the differentiability of the maps €I r-+ me imply that,
for every random sequence h n that is bounded in probability,
G n [ -/ii(m(}o+hn/..;n - m(}o) - h m(}o] O.
For nonrandom sequences h n this follows, because the variables have zero means, and vari-
ances that converge to zero, by the dominated convergence theorem. For general sequences
h n this follows from Lemma 19.31.
A second fact that we need and that is proved subsequent1y is the -Jfi-consistency of the
sequence en. By Corollary 5.53, the Lipschitz condition, and the twice differentiability of
the map €I r-+ P me, the sequence -Jfi (e n - €I) is bounded in probability.
The remainder of the proof is self-contained. In view of the twice differentiability of the
map €I r-+ P me, the preceding display can be rewritten as
( ) 1 -T - -T .
nIFn meo+hn/,.fn - meo == "2 h n Veohn + h n CGnmeo + op(l).
Because the sequence en is -Jfi-consistent, this is valid both for h n equal to h n == -Jfi (en - €lo)
and for h n == - V lCGnmeo' After simple algebrain the second case, we obtain the equations
( ) 1 AT A AT .
nIFn meo+hn/,.fn - meo == 2 hn Veohn + h n CGnmeo + op(l),
1
nlP' n (m(}o- v,,;;lGnmeo/..;n - m(}o) = - 2Gnm Ve;;-lGnm(}o + O p (1).
By the definition of en, the left side of the first equation is larger than the left side of the
second equation (up to op(l)) and hence the same relation is true for the right sides. Take
the difference, complete the square, and conclude that
(hn + V(} lGnm(}o)T V(}o(h n + V(} lGnm(}o) + ap(1) > O.
Because the matrix Veo is strict1y negative-definite, the quadratic form must converge to
zero in probability. The same must be true for II h n + Ve 1 CGnmeo II. .
The assertions of the preceding theorems must be in agreement with each other and
also with the informal derivati on leading to (5.20). If €I r-+ me (x) is differentiable, then a
maximizer of €I r-+ IF nme typically solves IF n 1/1'e == O for 1/1'e == me. Then the theorems and
(5.20) are in agreement provided that
a 2 a .
V e == ae 2 Pme == ae P1/1'e == P 1/1' e == Pme.
This involves changing the order of differentiation (with respect to €I) and integration (with
respect to x), and is usually permitted. However, for instance, the second derivative of Pme
may exist without €I r-+ me (x) being differentiable for all x, as is seen in the following
example.
5.24 Example (Median). The sample median maximizes the criterion function €I r-+
- 'L7 = 11 Xi - €I I. Assume that the distribution function F of the observations is differentiable
5.3 Asymptotic Normality
55
o
T"""
C\I
o
ex:>
o
<.o
o
o
o
o
-0.5
0.0
0.5
Figure 5.3. The distribution function of the sample median (dotted curve) and its normal approxi-
mation for a sample of size 25 from the Laplace distribution.
at its lnedian e o with positive derivative f (e o ). Then the sample median is asymptotically
normal.
This follows from Theorem 5.23 applied with me (x) == Ix - el - Ix I. As a consequence
of the triangle inequality, this function satisfies the Lipschitz condition with m (x) = 1.
Furthermore, the map e 1-+ me (x) is differentiable at e o except if x == e o , with meo (x) ==
-sign(x - (jo). By partial integration,
Pm() = ep(O) + [ (e - 2x) dP(x) - e(l - p(e)) = 2 [() P(x) dx - e.
J(o,e] lo
If F is sufficiently regular around e o , then Pme is twice differentiable with first derivative
2F(e) - 1 (which vanishes at e o ) and second derivative 2f(e). More generally, under the
minimal condition that F is differentiable at e o , the function Pme has a Taylor expansion
Pmeo + (e - ()0)22f(()0) + o(l() - ()012), so that we set Veo == 2f(eo). Because Pm o
== El == 1, the asymptotic variance of the median is 1/(2f(()0) )2. Figure 5.3 gives an
impression of the accuracy of the approximation. D
5.25 Example (Misspecified model). Suppose an experimenter postulates amodel {pe: ()
E e} for a sample of observations XI, . . . , X n' However, the model is misspecified in that
the true underlying distribution does not belong to the model. The experimenter decides to
use the postulated model anyway, and obtains an estimate en from maximizing the likelihood
L log Pe(X i ). What is the asymptotic behaviour of en?
At first sight, it might appear that en would behave erratically due to the use of the wrong
model. However, this is not the case. First, we expect that en is asymptotically consistent
for a value (jo that maximizes the function e 1---* p log pe, where the expectation is taken
under the true underlying distribution P. The density pe o can be viewed as the "projection"
56
M- and Z-Estimators
of the true underlying distribution P on the model using the Kullback-Leibler divergence,
which is defined as - P log(Pe / p), as a "distance" measure: Peo minimizes this quantity
over all densities in the model. Second, we expect that ,Jn(e n - ea) is asymptotically
normal with mean zero and covariance matrix
1 '.T 1
Ve p f,eof,eo Ve .
Here f,e == log Pe, and Veo is the second derivative matrix of the map e p log Pe. The
preceding theorem with me == log Pe gives sufficient conditions for this to be true.
The asymptotics give insight into the practical value of the experimenter' s estimate en.
This depends on the specific situation. However, if the model is not too far off from the truth,
then the estimated density Pen may be a reasonable approximation for the true density. D
5.26 Example (Exponentialfrailty model). Suppose that the observations are arandom
sample (XI, YI), ..., (X n , Y n ) of pairs of survival times. For instance, each Xi is the
survival time of a "father" and Yi the survival time of a "son." We assume that given
an unobservable value Zi, the survival times Xi and Yi are independent and exponentially
distributed with parameters Zi and e Zi, respectively. The value Zi may be different for each
observation. The problem is to estimate the ratio e of the parameters.
To fit this example into the i.i.d. set-up of this chapter, we assume that the values ZI, . . . , Zn
are realizations of arandom sample ZI, . . . , Zn from some given distribution (that we do
not have to know or parametrize).
One approach is based on the sufficiency of the variable Xi + e Yi for Zi in the case
that e is known. Given Zi == z, this "statistic" possesses the gamma-distribution with
shape parameter 2 and scale parameter Z. Corresponding to this, the conditional density of
an observation (X, Y) factorizes, for a given z, as he(x, y) ge(x + ey I z), for ge(s IZ) ==
z 2 se- zs the gamma-density and
e
he (x, y) == .
x +ey
Because the density of Xi + eY i depends on the unobservable value Zi, we might wish to
discard the factor ge (s I z) from the likelihood and use the factor he (x, y) only. Unfor-
tunately, this "conditionallikelihood" does not behave as an ordinary likelihood, in that
the corresponding "conditionallikelihood equation," based on the function he / he (x, y) ==
a / ae log he (x, y), does not have mean zero under e. The bias can be corrected by condi-
tioning on the sufficient statistic. Let
h ( h ) X - eY
o/e(X, Y) == 2e (X, Y) - 2eE e (X, Y) IX + eY == .
he he X + e Y
Next define an estimator en as the solution of JI» n o/e == O.
This works fairly nicely. Because the function e o/e (x, y) is continuous, and de-
creases strictly from 1 to -Ion (O, (0) for every x, y > O, the equation rP n o/e == O has a
unique solution. The sequence of solutions en can be seen to be consistent by Lemma 5.10.
By straightforward calculation, as e ---+ e a ,
e + e a 2e e a e a 1
Pe o/e == - - log - == -(ea - e) + o(e a - e).
o e - e a (e - e a ) 2 e 3e a
5.3 Asymptotic Normality
57
Rence the zero of e 1--+ Peo 1j;e is taken uniquely at e == e o . Next, the sequence -Jfi(e n - e o )
can be shown to be asymptotically normal by Theorem 5.21. In fact, the functions 1j; e (x, y)
are uniformly bounded in x, y > O and e ranging over compacta in (O, (0), so that, by the
mean value theorem, the function 1j; in this theorem may be taken equal to a constant.
On the other hand, although this estimator is easy to compute, it can be shown that it is
not asymptotically optimal. In Chapter 25 on semiparametric models, we discuss estimators
with a smaller asymptotic variance. O
5.27 Example (Nonlinear least squares). Suppose that we observe arandom sample (X 1,
YI), . . . , (X n, Y n ) from the distribution of a vec tor (X, Y) that follows the regression
model
Y == Jeo (X) + e,
E(e I X) == O.
Rere Je is a parametric family of regression functions, for instance Je (x) == el + e 2 ee 3 x,
and we aim at estimating the unknown vector e. (We assume that the independent variables
are arandom sample in order to fit the example in our i.i.d. notation, but the analysis could
be carried out conditionally as well.) The least squares estimator that minimizes
n
e f-* L (Yi - Je (Xi) ) 2
i=l
is an M-estimator for me(x, y) == (y - Je(x))2 (or rather minus this function). It should
be expected to converge to the minimizer of the limit criterion function
e 1--+ Pme == P (Jeo - Je)2 + Ee 2 .
Thus the least squares estimator should be consistent if 8 0 is identifiable from the model,
in the sense that 8 =f. 8 0 implies that Je (X) =f. Jeo (X) with positive probability.
For sufficiently regular regression models, we have
Pme p ((e - (0)T j e o )2 + Ee 2 .
This suggests that the conditions of Theorem 5.23 are satisfied with Veo == 2P je j; and
. o o
meo (x, y) == - 2(y - Jeo (x) ) Jeo (x). If e and X are independent, then this leads to the
asymptotic covariance matrix Ve 12Ee 2 . O
Besides giving the asymptotic normality of -Jfi(e n - e o ), the preceding theorems give
an asymptotic representation
A 1 -1 ( 1 )
en == e o + - Veo 1j; e o (X i) + o p r:; '
n i =l yn
If we neglect the remainder term, t then this means that en - e o behaves as the average of
the variables Ve 11j;e o (Xi)' Then the (asymptotic) "influence" of the nth observation on the
t To make the following derivati on rigorous, more information eonceming the remainder term would be necessary.
58
M - and Z -Estimatars
value of en can be computed as
1 1 n-I
en (XI, . . . , X n ) - e n - 1 (XI, . . . , X n - 1 ) - V e - 1 1f;e o (X n ) - " V e - 1 1f;e o (Xi)
n o n ( n - 1 ) o
z=1
1 -1 ( 1 )
== - V e 1f; eo (X n) + O p - .
n o n
Because the "influence" of an extra observation x is proportional to V e - 1 1f;e (x), the function
x V e - 1 1f;e (x) is called the asymptotic influence function of the estimator en. Influence
functions can be defined for many other estimators as well, but the method of Z -estimation
is particularly convenient to obtain estimators with given influence functions. Because Veo
is a constant (matrix), any shape of influence function can be obtained by simply choosing
the right functions 1f;e.
For the purpose of robust estimation, perhaps the most important aim is to bound the
influence of each individual observation. Thus, a Z -estimator is called B -robust if the
function 1f;e is bounded.
5.28 Example (Robust regression). Consider arandom sample of observations (X I, YI),
. . . , (X n , Y n ) following the linear regression model
Yi == el Xi + ei,
for i.i.d. errors el, . . . , en that are independent of XI, . . . , Xn. The classical estimator for the
regression parameter e is the least squares estimator, which minimizes :L7=1 (Yi - eT X i )2.
Outlying values of Xi ("leverage points") or extreme values of (Xi, Yi) jointly ("influence
points") can have an arbitrarily large influence on the value of the least-squares estimator,
which therefore is nonrobust. As in the case of location estimators, amore robu st estimator
for e can be obtained by replacing the square by a function m(x) that gro\vs less rapidly
as x ---+ 00, for instance m (x) == Ix I or m (x) equal to the primitive function of Huber' s 1/1'.
Usually, minimizing an expression of the type :L7=lm(Y i - eX i ) is equivalent to solving a
system of equations
n
L1f;(Y i - eTxi)X i == O.
i=l
Because E1f; (Y -el X)X == E1f; (e )EX, we can expect the resulting estimatorto be consistent
provided E 1/1' (e) == O. Furthermore, we should expect that, for Veo == E 1f;' (e) X X T ,
C A 1 -1 ( T )
vn(en-e o )== Veo 1f; yi-eoX i X i +o p (l).
n . I
l=
Consequently, even for a bounded function 1f;, the influence function (x, y) V e - l 1f; (y -
eT x)x may be unbounded, and an extreme value of an Xi may still have an arbitrarily
large influence on the estimate (asymptotically). Thus, the estimators obtained in this way
are protected against influence points but may still suffer from leverage points and hence
are only partly robu st. To obtain fully robust estimators, we can change the estimating
5.3 Asymptotic Norrnality
59
equations to
n
L1/1((Yi - eTXi)v(Xi))w(X i ) == O.
i=l
Here we protect against leverage points by choosing w bounded. For more fiexibility we
have also allowed a weighting factor v (Xi) inside 1/1. The choices 1/1 (x) == x, v (x) == 1 and
w (x) == x correspond to the (nonrobust) least-squares estimator.
The solution en of our final estimating equation should be expected to be consistent for
the solution of
o = E1jr((Y - eTX)v(X))w(X) = E1jr( (e + ei[ x - eT X)V(X))W(X).
If the function 1/1 is odd and the error symmetric, then the true value ea will be a solution
whenever e is symmetric about zero, because then E 1/1 (ea) == O for every a.
Precise conditions for the asymptotic normality of ,Jn(e n - e a ) can be obtained from
Theorems 5.21 and 5.9. The verification of the conditions of Theorem 5.21, which are "Iocai"
in nature, is relatively easy, and, if necessary, the Lipschitz condition can be relaxed by
using resu1ts on empirical processes introduced in Chapter 19 directly. Perhaps proving the
consistency of en is harder. The biggest technical problem may be to show that en == O p (1),
so it would help if e could a pri ori be restricted to a bounded set. On the other hand,
for bounded functions 1/1, the case of most interest in the present context, the functions
(x, y) 1/1 ((y - eT x) v (x) ) w (x) readily form a Glivenko-Cantelli class when e ranges
freely, so that verification of the strong uniqueness of e a as a zero becomes the main
challenge when applying Theorem 5.9. This leads to a combination of conditions on 1/1, v,
w, and the distributions of e and X. O
5.29 Example (Optimal robust estimators). Every sufficientlyregularfunction 1/1 defines
a location estimator en through the equation 'L7 = 11/1 (X i - e) == O. In order to choose among
the different estimators, we could compare their asymptotic variances and use the one with
the smallest variance under the postulated (or estimated) distribution P of the observations.
On the other hand, if we also wish to guard against extreme obervations, then we should
find a balance between robustness and asymptotic variance. One possibility is to use the
estimator with the smallest asymptotic variance at the postulated, ideal distribution Punder
the side condition that its infiuence function be uniformly bounded by some constant c. In
this example we show that for P the normal distribution, this leads to the Huber estimator.
The Z -estimator is consistent for the solution e a of the equation P 1/1 (. - e) == E 1/1 (X 1 -
e) == O. Suppose that we fix an underlying, ideal P whose "location" e a is zero. Then the
problem is to find 1/1 that minimizes the asymptotic variance p1/12j(p1/1')2 under the two
side conditions, for a given constant c,
1/1 (x)
su p < c and P"II' == O.
x P 1/1' - , 'f'
The problem is homogeneous in 1/1, and hence we may assume that P 1/1' == 1 without loss
of generality. Next, minimization of P 1/12 under the side conditions P 1/1 == O, P 1/1' == 1 and
111/1 1100 < C can be achieved by using Lagrange multipliers, as in problem 14.6 This leads
to minimizing
p1jr2 + AP1jr + JL(P1jr' - 1) = P( 1jr2 + 1jr(A + JL(p' jp)) - JL)
60
M - and Z-Estimators
for fixed "multipliers" A and f.-L under the side condition 111jJ 1100 < c with respect to 1jJ. This
expectation is minimized by minimizing the integrand pointwise, for every fixed x. Thus
the minimizing 1jJ has the property that, for every x separately, y == 1jJ (x) minimizes the
parabola y2 + AY + f.-Ly(p' / p)(x) over y E [-c, cJ. This readily gives the solution, with
[y] the value y truncated to the interval [c, d],
[ 1 1 p' ] c
1jJ(x) == --A - -f.-L-(x) .
2 2 p -c
The constants A and f.-L can be solved from the side conditions P1jJ == O and P1jJ' == 1. The
normal distribution P == <p has location score function p' / p (x) == - x, and by symmetry
it follows that A == O in this case. Then the optimal1jJ reduces to Huber' s 1/1' function. D
*5.4 Estimated Parameters
In many situations, the estimating equations for the parameters of interest contain prelim-
inary estimates for "nuisanee parameters." For example, many robust location estimators
are defined as the solutions of equations of the type
t V1 ( X; -: e ) == O.
i=l a
(5.30)
Here {j is an initial (robu st) estimator of scale, which is meant to stabilize the robustness
of the location estimator. For instance, the "cut-off" parameter k in Huber's 1/1'-function
determines the amount of robustness of Huber' s estimator, but the effect of a particular
choice of k on bounding the influence of outlying observations is relative to the range of
the observations. If the observations are concentrated in the interval [-k, k], then Huber's
1jJ yields nothing else but the sample mean, if all observations are outside [-k, k], we get
the median. Scaling the observations to a standard scale gives a clear meaning to the value
of k. The use of the median absolute deviation from the median (see. section 21.3) is often
recommended for this purpose.
If the scale estimator is itself a Z -estimator, then we can treat the pair (e, (j) as a Z-
estimator for a system of equations, and next apply the preceding theorems. More generally,
we can apply the following result. In this subsection we allow a condition in terms of
Donsker classes, which are discussed in Chapter 19. The proof of the following theorem
follows the same steps as the proof of Theorem 5.21.
5.31 Theorem. For each e in an open subset of"ffi.k and each 17 in a metric space, let x
1jJe,77(x) be an "ffi.k-valued measurablefunction such that the class offunctions {1/1'e,7] : Ile -
eoll < o, de17, 170) < o} is Donsker for some o > O, and such that PII1jJe,7] -1jJe O ,7]O 11 2 -+ O
as (e, 17) -+ (e o , 170). Assunte that P1jJe O ,7]O == O, and that the maps e P1jJe.7] are differ-
entiable at e o , uniformly in TJ in a neighborhood of 170 with nonsingular derivative lnatrices
" p
V eO ,7] such that V eo .7] -+ Veo, 770' If -J!iJP n 1jJe n ,fJl1 == op(l) and (en, r,n) -+ (e o , 170), then
-J!i(e n - e o ) == - Ve , o -J!i P1jJe o .fJn - Ve , o CG n 1jJe O ,7]O + o p (1 + -J!iIIP1/re o ,fJn II).
5.5 Maximum Likelihood Estimators
61
U nder the conditions of this theorem, the limiting distribution of the sequence -Jri (e n -
e o ) depends on the estimator fJn through the "drift" term -Jri p 1/1 e O,fln' In general, this gives
a contribution to the limiting distribution, and f}n must be chosen with care. If f}n is -Jri-
consistent and the map 17 r-+ P1/1e o ,rJ is differentiable, then the drift term can be analyzed
using the de1ta-method.
It may happen that the drift term is zero. If the parameters e and TJ are "orthogonal"
in this sense, then the auxiliary estimators f}n may converge at an arbitrarily slow rate and
affect the limit distribution of en only through their limiting value TJo.
5.32 Example (Symmetric location). Suppose that the distribution of the observations
is symmetric about e o . Let x r-+ 1/1 (x) be an antisymmetric function, and consider the
Z-estimators that solve equation (5.30). Because P1/1((X - eo)/a) == O for every a, by
the symmetry of P and the antisymmetry of 1/1, the "drift term" due to f} in the pre-
ceding theorem is identically zero. The estimator en has the same limiting distribu-
tion whether we use an arbitrary consistent estimator of a "true scale" ao or ao
itself. D
5.33 Example (Robust regression). In the linear regression model considered in Exam-
ple 5.28, suppose that we choose the weight functions V and w dependent on the data and
solve the robust estimator en of the regression parameters from
1 n
0== - L1/1((Yi - eTXi)Vn(Xi))Wn(Xi)'
n. 1
1=
This corresponds to defining a nuisance parameter 17 == (v, w) and setting 1/1e, v, w (x, y) ==
1/1 ( (y - eT x)v(x) )w(x). If the functions 1/1e,v,w run through a Donsker class (and they
easily do), and are continuous in (e, v, w), and the map e r-+ P1j;e,v,w is differentiable at
e o uniformly in (v, w), then the preceding theorem applies. If E1j;(ea) == O for every a,
then P1/1 e o,v,w == O for any v and w, and the limit distribution of -Jri(e n - e o ) is the same,
whether we use the random weight functions (v n , W n ) or their limit (vo, wo) (assuming that
this exists).
The purpose of using random weight functions could be, besides stabilizing the robu st -
ness, to improve the asymptotic efficiency of en. The limit (vo, wo) typically is not the
same for every underlying distribution P, and the estimators (un, W n ) can be chosen in such
away that the asymptotic variance is minimal. D
5.5 Maximum Likelihood Estimators
Maximum likelihood estimators are examples of M -estimators. In this section we special-
ize the consistency and the asymptotic normality results of the preceding sections to this
important special case. Our approach reverses the historicalorder. Maximum likelihood
estimators were shown to be asymptotically normal first by Fisher in the 1920s and rigor-
ously by Cramer, among others, in the 1940s. General M -estimators were not introduced
and studied systematically until the 1960s, when they became essential in the development
of robust estimators.
62
M - and Z - Estimators
If XI, . . . , X n are arandom sample from a density PFJ, then the maximum likelihood
estimator en maximizes the function e f--+ L log PFJ (Xi), or equivalently, the function
1 PFJ PFJ
Mn(e) == - IOg-(Xi) ==JIDnlog-.
n i=l PFJo PFJ o
(Subtraction of the "constant" L log PFJ o (Xi) turns out to be mathematically convenient.)
If we agree that log O == -00, then this expression is with probability 1 well defined if PFJ o
is the true density. The asymptotic function corresponding to M n ist
PFJ PFJ
M(e) == E FJo log -eX) == PFJo log-.
pFJo PFJ o
The number - M (e) is called the Kullback-Leibler divergence of PFJ and PFJ o ; it is often
considered a measure of "distance" between PFJ and PFJ o ' although it does not have the
properties of a mathematical distance. Based on the results of the previous sections, we
may expect the maximum likelihood estimator to converge to a point of maximum of M (e).
Is the true value e o always a point of maximum? The answer is affinnative, and, more over,
the true value is a unique point of maximum if the true measure is identifiable:
PFJ =I=- P FJO '
every e =I=- eo.
(5.34)
This requires that the model for the observations is not the same under the parameters e
and eo. Identifiability is a natural and even a necessary condition: If the parameter is not
identifiable, then consistent estimators cannot exist.
5.35 Lemma. Let {PFJ: e E e} be a collection of subprobability densities such that
(5.34) holds and such that P FJO is a probability measure. Then M(e) == PFJo log PFJ/ PFJ o
attains its maximum uniquely at eo.
Proof. First note that M (e o ) == PFJo log 1 == O. Rence we wish to show that M(e) is strictly
negative for e =I=- e o .
Because log x < 2(.JX - 1) for every x > O, we have, writing I-L for the dominating
measure,
PFJo log PFJ < 2P FJo ( J Pe - 1 ) == 2 J -J PFJ PFJo dJL - 2
PFJ o PFJo
< - J( - )2dfL.
(The last inequality is an equality if f PFJ dJL == 1.) This is always nonpositive, and is zero
only if Pe and PFJo are equal. By assumption the latter happens only if e == eo. .
Thus, under conditions such as in section 5.2 and identifiability, the sequence of maxi-
mum likelihood estimators is consistent for the true parameter.
t Presently we take the expeetation Peo under the parameter eo, whereas the derivati on in seetion 5.3 is valid for a
generic underlying probability structure and does not coneeptually require that the set of parameters e indexes
a set of underlying distributions.
5.5 Maximum Likelihood Estimators
63
This conclusion is derived from viewing the maximum likelihood estimator as an M-
estimator for me == log Pe. Sometimes it is technically advantageous to use a different
starting point. For instance, consider the function
I Pe + peo
me == og .
2p e o
By the eoncavity of the logarithm, the maximum likelihood estimator e satisfies
1 Pe 1
IfDnme > IfD n -log - + IfD n -log 1 > O == Pnmeo.
2 peo 2
Even though e does not maximize () 1---+ IfD nme, this inequality can be used as the starting
point for a consistency proof, since Theorem 5.7 requires that M n (e) > M n (()a) - o p (1)
only. The true parameter is still identifiable from this criterion function, because, by the
preceding lemma, Pe o me == O implies that (Pe + pe o ) 12 == pe o ' or Pe == pe o ' A technical
advantage is that me > log(1/2). For another variation, see Example 5.17.
Consider asymptotic normality. The maximum likelihood estimator solves the likelihood
equations
a n
- L log Pe(X i ) == O.
a() i=l
Renee it is a Z-estimator for 1fe equal to the score function £e == al a() log Pe of the model.
In view of the results of section 5.3, we expect that the sequence -Vfi(e n - ()) is, under (),
asymptotically normal with mean zero and eovarianee matrix
(Pe l e)-l Pefef (Pei ) -1.
(5.36)
Under regularity eonditions, this reduees to the inverse of the Fisher information matrix
. 'T
le == Pe£e£e .
To see this in the case of a one-dimensional parameter, differentiate the identity J pe d J.L - 1
twice with respect to (). Assuming that the order of differentiation and integration can be
reversed, we obtain J pe d J.L = J pe d J.L = O. Together with the identities
. pe
£e == -;
pe
.. _ pe ( Pe ) 2
£e - - - -
pe pe
this implies that Pele == O (scores have mean zero), and Peie == - le (the curvature of the
likelihood is equal to minus the Fisher information). Consequently, (5.36) reduces to le-I.
The higher-dimensional case follows in the same way, in which we should interpret the
identities Pele == O and Peie == - le as a veetor and a matrix identity, respeetively.
We conelude that maximum likelihood estimators typieally satisfy
r:: A e ( -1 )
Vn(()n - ()) 'v'-t N O, le .
This is a very important result, as it implies that maximum likelihood estimators are asymp-
totieallyoptimal. The eonvergenee in distribution means roughly that the maximum likeli-
hood estimator en is N ((), (n1e)-1 )-distributed for every (), for large n. Renee, it is asymp-
totieally unbiased and asymptotieally of varianee (nle)-l. Aeeording to the Cramer-Rao
64
M - and Z-Estimators
theorem, the variance of an unbiased estimator is at least (n le) -1. Thus, we could in-
fer that the maximum likelihood estimator is asymptotically uniformly minimum-variance
unbiased, and in this sense optimal. We write "could" because the preceding reasoning is
informal and unsatisfying. The asymptotic norrnality does not warrant any conclusion about
the convergence of the moments Ee en and vare en; we have not introduced an asymptotic
version of the Cramer-Rao theorem; and the Cramer-Rao bound does not make any assertion
concerning asymptotic normality. Moreover, the unbiasedness required by the Cramer-Rao
theorem is restrictive and can be relaxed considerably in the asymptotic situation.
However, the message that maximum likelihood estimators are asymptotically efficient
is correct. We give a precise discussion in Chapter 8. The justification through asymptotics
appears to be the only general justification of the method of maximum likelihood. In some
form, this result was found by Fisher in the 1920s, but a better and more general insight
was only obtained in the period from 1950 through 1970 through the work of Le Cam and
others.
In the preceding informal derivations and discussion, it is implicitly understood that the
density pe possesses at least two derivatives with respect to the parameter. Although this
can be relaxed considerably, a certain amount of smoothness of the dependence () p() is
essential for the asymptotic norrnality. Compare the behavior of the maximum likelihood
estimators in the case ofuniforrnly distributed observations: They are neither asymptotically
normal nor asymptotically optimal.
5.37 Example (Uniform distribution). Let XI, . . . , X n be a sample from the uniform
distribution on [O, ()]. Then the maximum likelihood estimator is the maximum X (n) of the
observations. Because the variance of X(n) is of the order O(n- 2 ), we expect that a suitable
norming rate in this case is not -J1i, but n. Indeed, for each x < O
( ) ( X ) n ( e +x/n ) n xj()
p () n (X (n) - e) < x == P () XI < e + n == e ---+ e .
Thus, the sequence -n(X(n) - ()) converges in distribution to an exponential distribution
with mean e. Consequently, the sequence -J1i (X (n) - e) converges to zero in probability.
N ote that most of the informal operations in the preceding introduction are illegal or not
even defined for the uniform distribution, starting with the definition of the likelihood equa-
tions. The informal conclusion that the maximum likelihood estimator is asymptotically
optimal is also wrong in this case; see section 9.4. D
We conclude this section with a theorem that establishes the asymptotic normality of
maximum likelihood estimators rigorously. Clearly, the asymptotic normality follows from
Theorem 5.23 applied to m() == log p(), or from Theorem 5.21 applied with 1j;() == £() equal
to the score function of the lTIodel. The following result is aminor variation on the first
theorem. Its conditions somehow also ensure the relationship P()f() == - l() and the twice-
differentiability of the map () p()o log p(), even though the existence of second derivatives
is not part of the assumptions. This remarkable phenomenon results from the trivial fact
that square roots of probability densities have squares that integrate to 1. To exploit this,
we require the differentiability of the maps e ffi, rather than of the maps e log p().
A statistical model (P(): e E 8) is called differentiable in quadratic mean if there exists a
5.5 Maximum Likelihood Estimators
65
measurable vector-valued function £eo such that, as e -+ e o ,
f [-Jii8 - .;pa; - (e - eol £eo.;pa; r dJ-L = o(lIe - e o I1 2 ),
(5.38)
This property also plays an important role in asymptotic optimality theory. A discussion,
including simple conditions for its validity, is given in Chapter 7. It should be noted that
ala l ( a )
ae -Jii8 = 2-Jii8 ae Pe = 2: ae log Pe -Jii8.
Thus, the function £e o in the integral really is the score function of the model (as the
notation suggests), and the expression leo == Peo.€eo.€ defines the Fisher information matrix.
However, condition (5.38) does not require existence of alae pe(x) for every x.
5.39 Theorem. Suppase that the model (Pe: e E 8) is differentiable in quadratic me an
at an inner point e o of 8 c IR k . Furthermore, suppose that there exists a measurable
function .€ with Peog2 < 00 such that, for every el and e 2 in a neighborhood of e o ,
Ilog P e 1 (x) - log P e 2 (x) I < i(x) Ile I - e 2 11.
lf the Fisher information matrix leo is nonsingular and en is consistent, then
A -1 1 L: n .
Vii(e n - e o ) == le Vii feo(Xi) +oP e (1).
o n o
i=I
In particular, the sequence ,Jn(e n - e o ) is asymptotically normal with mean zero and
covariance matrix le 1.
*ProoJ. This theorem is acorollary of Theorem 5.23. We shalI show that the conditions
of the latter theorem are satisfied for me == log pe and Veo == - leo'
Fix an arbitrary converging sequence of vectors h n -+ h, and set
W n == 2 ( Peo+hn/,Jn - 1 ) .
peo
By the differentiability in quadratic mean, the sequence ,JnW n converges in L 2 (Peo) to the
function h T .€e o . In particular, it converges in probability, whence by adelta method
vii (10 g P eo +h n /,Jn - log P e o ) == 2Vii lo g (1 + i W n) h T ge o .
In view of the Lipschitz condition on the map e 1---+ log pe, we can apply the dominated-
convergence theorem to strengthen this to convergence in L 2 (Pe o )' This shows that the map
e 1---+ log Pe is differentiable in probability, as required in Theorem 5.23. (The preceding
argument considers only sequences en of the speci al for m e o + h n 1,Jn approaching e o .
Because h n can be any converging sequence and -J n + 1/,Jn -+ 1, these sequences are
actuall y not so special. By re- indexing the result can be seen to be true for any en -+ e o .)
Next, by computing means (which are zero) and variances, we see that
CG n [Vii (log Peo+h,,/ -log peo) - h T £e o ] O.
66
M - and Z-Estimators
Equating this result to the expansion given by Theorem 7.2, we see that
nPeo (log Peo+h,z/ft -log peo) -+ _ hT Ie o h .
Hence the map e 1--+ Pe o log Pe is twice-differentiable with second derivative matrix - leo,
or at least permits the corresponding Taylor expansion of order 2. .
5.40 Example (Binary regression). Suppose that we observe arandom sample (XI,
YI), . . . , (X n , Y n ) consisting of k-dimensional vectors of "covariates" Xi, and 0-1 "response
variables" Yi, following the model
Pe(Y i == 11 Xi == x) == \II (eT x).
Here \II: JR 1--+ [O, 1] is a known continuously differentiable, monotone function. The choices
\II(e) == 1/(1 +e-e) (the logistic distribution function) and \II == <p (the normal distribution
function) correspond to the logit model and probit model, respectively. The maximum
likelihood estimator en maximizes the (conditional) likelihood function
n n
e 1--+ Il pe (Yi I Xi): == Il \II (eT Xi) Yi (1 - \II ( eT Xi) ) 1- Yi .
i=l i=I
The consistency and asymptotic normality of en can be proved, for instance, by combining
Theorems 5.7 and 5.39. (Altematively, we may follow the classical approach given in sec-
tion 5.6. The latter is particularly attractive for the logit model, for which the log likelihood
is strictly concave in e, so that the point of maximum is unique.) For identifiability of e we
must assume that the distribution of the Xi is not concentrated on a (k - l)-dimensional
affine subspace of JRk. For simplicity we assume that the range of Xi is bounded.
The consistency can be proved by applying Theorem 5.7 with me == log(Pe + Pe o )/2.
Because peo is bounded away from O (and (0), the function me is somewhat better behaved
than the function log Pe.
By Lemma 5.35, the parameter e is identifiable from the density Pe. We can redo the
proof to see that, with ,:s meaning "less than up to a constant,"
( 1/2 ) 2
PeD(me -meD);:S - f ( (pe + pe D )) - p :2 dfL
;:s -E('l1(e T X) - 'l1(elX)r.
This shows that e o is the unique point of maximum of e 1--+ Pe o me. Furthermore, if Pe o mek -+
Peomeo, then el X el x. Ifthe sequence e k is also bounded, then E( (e k - eo)T X)2 -+ O,
whence ()k 1---+ e o by the nonsingularity of the matrix EX X T . On the other hand, II()k II cannot
have a diverging subsequence, because in that case el/II ek II X o and hence e k I II e k II -+ O
by the same argument. This verifies condition (5.8).
Checking the uniform convergence to zero of sUPe IIfDnme - Pme I is not trivial, but
it becomes an easy exercise if we employ the Glivenki -Cantelli theorem, as discussed in
Chapter 19. The functions x 1---+ \II (eT x) form a VC-class, and the functions Ine take the
form me (x, y) == <p (\II (e T x), y, \II (el x»), where the function <p (y, y, 1]) is Lipschitz in its
first argument with Lipschitz constant bounded above by 1/ rJ + 1/ (1 - 17). This is enough to
5.6 Classical Conditions
67
ensure that the functions me form a Donsker class and hence certainly a Glivenko-Cantelli
class, in view of Example 19.20.
The asymptotic normality of vn(e n - e) is now a consequence of Theorem 5.39. The
score function
Y - \li ( eT X )
i ( I ) . \llI ( e T )
e Y X == \lI(e T x)(l _ \li) (eT x) X X
is uniformly bounded in x, y and e ranging over compacta, and continuous in e for every
X and y. The Fisher information matrix
\li' (e T X)2
I ==E XX T
e \li (eT X)(l - \li) (eT X)
is continuous in e, and is bounded below by a multiple of EX X T and hence is nonsingular.
The differentiability in quadratic mean follows by calculus, or by Lemma 7.6. O
*5.6 Classical Conditions
In this section we discuss the "classical conditions" for asymptotic normality of M -estima-
tors. These conditions were formulated in the 1930s and 1940s to make the informal deriva-
tions of the asymptotic normality of maximum likelihood estimators, for instance by Fisher,
mathematically rigorous. Although Theorem 5.23 requires less than a first derivative of
the criterion function, the "classical conditions" require existence of third derivatives. It
is clear that the classical conditions are too stringent, but they are still of interest, because
they are simple, lead to simple proofs, and nevertheless apply to many examples. The
classical conditions also ensure existence of Z -estimators and have a little to say about their
consistency.
We describe the classical approach for general Z-estimators and vector-valued parame-
ters. The higher-dimensional case requires more skill in calculus and matrix algebra than
is necessary for the one-dimensional case. When simplified to dimension one the argu-
ments do not go much beyond making the informal derivati on leading from (5.18) to (5.19)
rlgorous.
Let the observations XI, . . . , X n be a sample from a distribution P, and consider the
estimating equations
1 n
\lIn(e) == - Ll/1e(X i ) == IPnl/1e,
n. 1
l=
\lI(e) == Pl/1e.
The estimator e 11 is a zero of \li n, and the true value e a a zero of \li. The essential condition
of the following theorem is that the second-order p arti al derivatives of l/1e (x) with respect
to e exist for every x and satisfy
a 2 l/1eh(X) ..
ae;e j < 1jJ(x),
for some integrable measurable function 1/1. This should be true at least for every e in a
neighborhood of e a .
68
M - and Z-Estimators
5.41 Theorem. For each 8 in an open subset of Euclidean space, let 8 H- O/e (x) be
twice continuously differentiable for every x. Suppose that P o/eo == O, that P Ilo/eo 11 2 < 00
and that the matrix PO/ eo exists and is nonsingular. Assume that the second-order partial
derivatives are dominated by afixed integrable function 1f (x) for every 8 in a neighborhood
of80' Then every consistent estimator sequence en such that \11 n (e n) == O for every n satisfies
C A ( . ) -1 1 f-.
yn(8n - 8 0 ) == - Po/e o vn o/eo(Xi) + op(l).
n. 1
l=
In particular, the sequence vn(e n - 8 0 ) is asymptotically normal with mean zero and
covariance matrix (P e o ) -1 P o/eo 0/ (P e o ) -1.
Proof. By Taylor' s theorem there exists a (random) vec tor en on the line segment between
8 0 and en such that
A . Al" T" - A
O == \11 n (8 n) == \li n (8 0 ) + \li n (8 0 ) ( 8 n - 8 0 ) + 2 (8 n - 8 0 ) \li n (8 n) ( 8 n - 8 0 ).
The first term on the right \lin (8 0 ) is an average of the i.i.d. random vectors o/e o (Xi), which
have mean Po/eo == O. By the central limit theorem, the sequence vn'1!n (8 0 ) converges
in distribution to a multivariate normal distribution with mean O and covariance matrix
Po/eo o/l. The derivative n (8 0 ) in the second term is an average also. By the law of
o .
large numbers it converges in probability to the matrix V == po/ eo. The second derivative
W n (e n) is a k - vector of (k x k) matrices depending on the second -order derivatives 1f e. By
assumption, there exists a ball B around 8 0 such that 1f e is dominated by 111/1 II for every
8 E B. The probability of the event {enE B} tends to 1. On this event
II " - II 1 f-. .. 1 f-. II .. II
\li n (8 n ) == - 0/ en (Xi) < - 0/ (Xi) .
n i=l n i=l
This is bounded in probability by the law of large numbers. Combination of these facts
allows us to rewrite the preceding display as
- \lin (8 0 ) == (v + O p (1) + (e n - 8 0 ) O p (1) ) (e n - 8 0 ) == (V + O p (1) ) (e n - 8 0 ),
because the sequence (en - 80) O p (1) == o p (1) O p (1) converges to O in probability if en
is consistent for 8 0 . The probability that the matrix Veo + o p (1) is invertible tends to 1.
Multiply the preceding equation by vn and apply (V + op(l) )-ll e ft and right to complete
the proof. .
In the preceding sections, the existence and consistency of solutions en of the estimating
equations is assumed from the start. The present smoothness conditions actually ensure the
existence of solutions. (Again the conditions could be significant1y relaxed, as shown in
the next proof. ) Moreover, provided there exists a consistent estimator sequence at all, it is
always possible to select a consistent sequence of solutions.
5.42 Theorem. Under the conditions of the preceding theorem, the probability that the eq-
uation IP n o/e == O has at least one root tends to 1, as n 00, and there exists a sequence
of roots en such that en 8 0 in probability. If 1/Je == me is the gradient of some function
5.6 Classical Conditions
69
me and 8 0 is a point of loeal maximum of 8 1---+ P me, then the sequenee en can be ehosen
to be Zoeal maxima of the maps 8 1---+ IID nme.
Proof. Integrate the Taylor expansion of 8 1---+ 1/1e (x) with respect to x to find that, for a
point e == e (x) on the line segment between 8 0 and 8,
. 1 T"
P 1/1 () == p 1/1 eo + p 1/1 eo (8 - ( 0 ) + 2: (8 - ( 0 ) P 1/1 e (8 - ( 0 ).
By the domination condition, II P1f e II is bounded by p 111f II < 00 if 8 is sufficiently close
to 8 0 . Thus, the map \¥ (8) == P1/1e is differentiable at 8 0 . By the same argument \Il is
differentiable throughout a small neighborhood of 8 0 , and by a similar expansion (but now
to first order) the derivative P 1/1 e can be seen to be continuous throughout this neighborhood.
Because P 1/1 eo is nonsingular by assumption, we can make the neighborhood still smaller,
if necessary, to ensure that the derivative of \Il is nonsingular throughout the neighborhood.
Then, by the inverse function theorem, there exists, for every sufficiently small 8 > O, an
- -
open neighborhood G 8 of 8 0 such that the map \Il: Go 1---+ ball (O, 8) is a homeomorphism.
The diameter of G 8 is bounded by a multiple of 8, by the mean-value theorem and the fact
that the norms of the derivatives (P 'if;- e) -1 of the inverse \Il -1 are bounded.
Combining the preceding Taylor expansion with a similar expansion for the sample
version \Iln (8) == JP n 1/1e, we see
s up IIW n (8) - \11(8) II < op(l) + 80 p (1) + 8 2 0 p (1),
eEG 8
where the op(l) term s and the Op(l) term result from the law of large numbers, and are
uniform in small 8. Because P( o p (1) + 80 p (1) > 8) --+ O for every 8 > O, there exists
8n t O such that P(op(l) + 8 n o p (1) > 8n) --+ O. If Kn,o is the event where the left side of
the preceding display is bounded above by 8, then P(Kn,on) --+ 1 as n --+ 00.
On the event Kn,o the map 8 1---+ 8 - \Iln o \Il-I (8) maps ball (O, 8) into itself, by the
definitions of G 8 and Kn,o. Because the map is also continuous, it possesses a fixed-point
- -
in ball (O, 8), by Brouwer's fixed point theorem. This yields a zero of \lin in the set Go,
whence the first assertion of the theorem.
For the final assertion, first note that the Hessian P 1/1 e o of 8 1---+ P me at 8 0 is negative-
definite, by assumption. A Taylor expansion as in the proof of Theorem 5.41 shows that
. . P A P .
IIDn1/1en - IIDn1/1eo --+ O for every 8n --+ 8 0 . Hence the Hessian IPn1/1en of 8 1---+ IPnme at any
consistent zero en converges in probability to the negative-definite matrix P 'if;- eo and is
negative-definite with probability tending to 1. .
The assertion of the theorem that there exists a consistent sequence of roots of the
estimating equations is easily misunderstood. It does not guarantee the existence of an
asymptotically consistent sequence of estimators. The only claim is that a clairvoyant
statistician (with preknowledge of ( 0 ) can choose a consistent sequence of roots. In reality,
it may be impossible to choose the right solutions based only on the data (and knowledge
of the model). In this sense the preceding theorem, a standard result in the literature, looks
better than it is.
The situation is not as bad as it seems. One interesting situation is if the solution of the
estimating equation is unique for every n. Then our solutions must be the same as those of
the clairvoyant statistician and hence the sequence of solutions is consistent.
70
M - and Z -Estimators
In general, the deficit can be repaired with the help of a preliminary sequence of estimators
en. If the sequence en is consistent, then it works to choose the root en of IP n 1jJe == O that
.- "',.., AC ,..,
is closest to () n' Because II () n - () n II is smaller than the distance II () n - () n II between the
clairvoyant sequence e and en, both distances converge to zero in probability. Thus the
sequence of closest roots is consistent.
The assertion of the theorem can also be used in anegative direction. The point ()a in
the theorem is required to be a zero of () 1---+ P 1jJe, but, apart from that, it may be arbitrary.
Thus, the theorem implies at the same time that a malicious statistician can always choose
a sequence of roots en that converges to any given zero. These may inc lude other points
besides the "true" value of (). Furthermore, inspection of the pro of shows that the sequence
of roots can also be chosen to jump back and forth between two (or more) zeros. If the
function () 1---+ P 1jJe has multiple roots, we must exercise care. We can be sure that certain
roots of () 1---+ IP n 1jJe are bad estimators.
Part of the problem here is caused by using estimating equations, rather than maximiza-
tion to find estimators, which blurs the distinction between points of absolute maximum,
local maximum, and even minimum. In the light of the results on consistency in section 5.2,
we may expect the location of the point of absolute maximum of () 1---+ IP nme to converge
to a point of absolute maximum of () 1---+ P me. As long as this is unique, the absolute
maximizers of the criterion function are typically consistent.
5.43 Example (Weibull distribution). Let XI, . . . , X n be a sample from the Weibull dis-
tribution with density
P ( x ) == xe-Ie-Xe ja
e,a ,
a
x > O, () > O, a > O.
(Then a lje is a scale parameter.) The score function is given by the partial derivatives of
the log density with respect to () and a:
. ( 1 x e
fe,a(x) == () + logx - logx,
The likelihood equations I: £e,a (xi) == O reduce to
w _ 2- + Xe ) .
a a 2
1 n
a == - '"""' x .
l'
n. I
l=
1 1 I: 'l=lxr log Xi
() + - L log Xi - I: n e = O.
n. I . I X.
l= l= l
The second equation is strictly decreasing in (), from 00 at () == O to log x -log x(n) at () == 00.
Rence a solution exists, and is unique, unless all Xi are equal. Provided the higher-order
derivatives of the score function exist and can be dominated, the sequence of maximum
likelihood estimators (en, cTn) is asymptotically normal by Theorems 5.41 and 5.42. There
exist four different third-order derivatives, given by
a 3 £e,a (x)
a()3
a 3 fe,a (x)
a()2a a
a 3 £e,a (x)
2 x e
== - - - log3 X
()3 a
x e
== 2log2x
a
2x e
== --logx
a()aa 2 a 3
a 3 £e,a(x) 2 6x e
== -- + -
aa 3 a 3 a 4 .
5.7 One-Step Estimators
71
For e and (5 ranging over sufficiently small neighborhoods of e o and (50, these functions are
dominated by a function of the form
M(x) == A(I + xB)(I + Ilogxl + ... + IlogxI 3 ),
for sufficiently large A and B. Because the Weibull distribution has an exponentially small
tail, the mixed moment Eeo,croXPllog X/q is finite for every p, q > O. Thus, all moments of
le and 'ie exist and M is integrable. O
*5.7 One-Step Estimators
The method of Z-estimation as discussed so far has two disadvantages. First, it may be
hard to find the roots of the estimating equations. Second, for the roots to be consistent,
the estimating equation needs to behave well throughout the parameter set. For instance,
existence of a second root cIo se to the boundary of the parameter set may cause trouble. The
one-step method overcomes these problem s by building on and improving a preliminary
estimator 8 n .
The idea is to solve the estimator from a line ar approximation to the original estimating
equation W n (e) == O. Given a preliminary estimator 8 n, the one-step estimator is the solution
(in e) to
W n (8n) + *n(8n)(e - 8n) == O.
This corresponds to replacing W n (e) by its tangent at 8 n, and is known as the method of
Newton-Rhapson in numerical analysis. The solution e == en is
" - . - -1 -
en == en - wn(e n ) wn(e n ).
In numerical analysis this procedure is iterated a number of times, taking en as the new
preliminary guess, and so on. Provided that the starting point 8 n is well chosen, the sequence
of solutions converges to a root of W n. Our interest here goe s in a different direction. We
suppose that the preliminary estimator 8 n is already within range n -1/2 of the true value
of e. Then, as we shall see, just one iteration of the Newton-Rbapson scheme produces
an estimator en that is as good as the Z -estimator defined by W n. In fact, it is better in
that its consistency is guaranteed, whereas the true Z -estimator may be inconsistent or not
uniquely defined.
In this way consistency and asymptotic normality are effectively separated, which is
useful because these two aims require different properties of the estimating equations.
Good initial estimators can be constructed by ad-hoc methods and take care of consistency.
N ext, these initial estimators can be improved by the one-step method. Thus, for instance,
the good properties of maximum likelihood estimation can be retained, even in cases in
which the consistency fails.
In this section we impose the following condition on the random criterion functions W n .
For every constant M and a given nonsingular matrix wo,
sup 1101(W n (8) - W n (8 0 )) - 001(8 - 80) II O.
-Ji1lle-e o II <M
(5.44 )
Condition (5.44) suggests that w n is differentiable at e o , with derivative tending to \Ilo, but
this is not an assumption. We do not require that a derivative w n exists, and introduce
72
M- and Z-Estimators
a further refinement of the Newton-Rhapson scheme by replacing W n (en) by arbitrary
. .
estimators. Given nonsingular, random matrices \l1n,o that converge in probability to \l10
define the one-step estimator
A - . -1 -
(jn == (jn - \l1n,oW n ((jn).
Call an estimator sequence en -Jn-consistent if the sequence -Jn(e n - (jo) is uniformly
tight. The interpretation is that en already determines the value (jo within n- 1 / 2 -range.
5.45 Theorem (One-step estimation). Let -JnW n ((jo) Z and let (5.44) hold. Then the
one-step estimator en, for a given -Jn-consistent estimator sequence en and estimators
. p.
\IIn,o-+ Wo, satisfies
(en - (jo) == _ Ol \l1n((jO) + op(l).
5.46 Addendum. For Wn((j) == JPno/ e condition (5.44) is satisfied under the conditions of
. . .
Theorem 5.21 with Wo == Veo, and under the conditions of Theorem 5.41 with \Ilo == PO/ eo.
Proof. The standardized estimator n,O-Jn(( n - (jo) equals
n,o (en - (jo) - (\IIn(en) - \lin ((jo)) - W; Wn((jO)'
By (5.44) the second term can be replaced by - o-Jn(en -(jo) +0 p (1). Thus the expression
can be rewritten as
( n,O - Wo) (en - (jo) - Wn((jO) + op(l).
The first term converges to zero in probabi1ity, and the theorem follows after application of
Slutsky's lemma.
For a proof of the addendum, see the proofs of the corresponding theorems. .
If the sequence -Jn (e n - (jo) converges in distribution, then it is certainly uniformly tight.
Consequently, a sequence of one-step estimators is -Jn-consistent and can itself be used as
preliminary estimator for a second iteration of the modified Newton-Rhapson algorithm.
Presumably, this would give a value closer to a root of W n . However, the limit distribution
of this "two-step estimator" is the same, so that repeated iteration does not give asymptotic
improvement. In practice a multistep method may nevertheless give better results.
We close this section with a discussion of the džscretžzation trick. This device is most1y
of theoretical value and has been introduced to relax condition (5.44) to the following. For
every nonrandom sequence (jn == (jo + O(n- 1 / 2 ),
!! (\¥n(en) - \¥n(eo») - q,o (en - e o )!! o.
(5.47)
This new condition is less stringent and much easier to check. It is sufficient1y strong if
the preliminary estimators en are discretized on grids of mesh width n -1/2. For instance,
en is suitably discretized if all its realizations are points of the grid n- 1 / 2 1/ (consisting
of the points n- 1 / 2 (ž1, . . . , ik) for integers il, . . . , Žk). This is easy to achieve, but perhaps
unnatural. Any preliminary estimator sequence en can be discretized by replacing its values
5.7 One-Step Estimators
73
by the closest points of the grid. Because this changes each coordinate by at most n -1/2,
,Jn-consistency of en is retained by discretization.
Define a one-step estimator en as before, but now use a discretized version of the pre-
liminary estimator.
5.48 Theorem (Discretized one-step estimation). Let ,Jnwn(e o ) Z andlet(5.47)hold.
Then the one-step estimator en, for a given ,Jn-consistent, discretized estimator sequence
- . p.
en and estimators Wn,o---+ WO, satisjies
A . 1
,Jn ( en - e o ) == - W O- ,Jn W n (e O ) + O p ( 1 ) .
5.49 Addendum. For \lin (e) == TID n O/e and TID n the empirical measure of arandom sample
from a density pe that is differentiable in quadratic mean (5.38), condition (5.47), is satisjied,
. .T
with \IJo == - Peo 1/J e o£e o ' if, as e ---+ e o ,
f [1/rev0JO - 1/re o JP80]2 dJL --+ O.
Proof. The arguments of the previous proof apply, except that it must be shown that
R ( en) : == -Jn ( W n (tj n) - W n ( e o )) - W O- 1 -Jn (en - eo)
converges to zero in probability. Fix 8 > O. By the ,Jn-consistency, there exists M with
p(,Jnlle n - eoll > M)< 8. If ,Jnlle n - eoll < M, then en equals one of the values in the
set Sn == {e E n- 1 / 2 Z k : Ile - eo/l < n- 1 / 2 M}. For each M and n there are only finitely
many elements in this set. Moreover, for fixed M the number of elements is bounded
independently of n. Thus
p(IIR(en)1I > 8) < 8 + L p(/lR(en)/I > 8 /\ en == en)
en E Sn
< 8 + L p(/lR(en)11 > 8).
en ES n
The maximum of the terms in the sum corresponds to a sequence of nonrandom vectors en
with en == e o + O (n- 1 / 2 ). It converges to zero by (5.47). Because the number of term s in
the sum is bounded independently of n, the sum converges to zero.
For a proof of the addendum, see proposition A.1 O in [139]. .
If the score function £e of the model also satisfies the conditions of the addendum,
then the estimators Wn,o == - Pen 1/Je n .e l are consistent for W o . This shows that discretized
one-step estimation can be carried through under very mild regularity conditions. Note
that the addendum requires only continuity of e !---+ O/e, whereas (5.47) appears to require
differentiability.
5.50 Example (Cauchy distribution). Suppose XI, . . . , X n are a sample from the Cauchy
location family pe(x) == n- 1 (1 + (x - e)2)-1. Then the score function is given by
. 2(x -e)
£ ( x ) ==
e 1 + (x - e)2 .
74
M - and Z -Estimators
o
o
I
o
LO
I
o
o
N
I
o
LO
";J
o
o
c?
-200
-100
o
100
Figure 5.4. Cauchy log likelihood function of a sample of 25 observations, showing three local
maxima. The value of the absolute maximum is well-separated from the other maxima, and its
location is close to the true value zero of the parameter.
This function behaves like 1/ x for x ---+ :1::00 and is bounded in between. The second
moment of fe (X 1) therefore exists, unlike the moments of the distribution itself. Because
the sample mean possesses the same (Cauchy) distribution as asingle observation XI, the
sample mean is a very inefficient estimator. Instead we could use the median, or another
M -estimator. However, the asymptotically best estimator should be based on maximum
likelihood. We have
. 4(x - e)((x - e)2 - 3)
fe(x) == .
(1 + (x - e)2)3
The tails of this function are of the order 1/ x 3 , and the function is bounded in between.
These bounds are uniform in e varying over a compact interval. Thus the conditions of
Theorems 5.41 and 5.42 are satisfied. Since the consistency follows from Example 5.16,
the sequence of maximum likelihood estimators is asymptotically normal.
The Cauchy likelihood estimator has gained a bad reputation, because the likelihood
equation L le (Xi) == O typicall y has several roots. The number of roots behaves asymp-
totically as two times a Poisson(l/n) variable plus 1. (See [126].) Therefore, the one-step
(or possibly multi-step method) is often recommended, with, for instance, the median as the
initial estimator. Perhaps a better solution is not to use the likelihood equations, but to de ter-
mine the maximum likelihood estimator by, for instance, visual inspection of a graph of the
likelihood function, as in Figure 5.4. This is particularly appropriate because the difficulty of
multiple roots does not occur in the two parameter location-scale model. In the model with
density Pe (x / a) / a , the maximum likelihood estimator for (e, a) is unique. (See [25].) O
5.51 Example (Mixtures). Let f and g be given, positive probability densities on the real
line. Consider estimating the parameter e == (11-, V, a, T, p) based on arandom sample from
5.8 Rates of Convergence
75
the mixture density
( X-f.-L ) 1 ( X-V ) 1
X pf a a + (1 - p)g i i .
If f and g are sufficiently regular, then this is a smooth five-dimensional parametric model,
and the standard theory should apply. Unfortunately, the supremum of the likelihood over
the natural parameter space is 00, and there exists no maximum likelihood estimator. This
is seen, for instance, from the fact that the likelihood is bigger than
( Xl-f.-L ) 1 O n ( Xi-V ) 1
pf - (1 - p)g -.
a a i=2 i i
If we set f.-L == XI and next maximize over a > O, then we obtain the value 00 whenever
p > O, irrespective of the values of V and i.
A one-step estimator appears reasonable in this example. In view of the smoothness of
the likelihood, the general theory yields the asymptotic efficiency of a one-step estimator
if started with an initial ,Jn-consistent estimator. Moment estimators could be appropriate
initial estimators. D
*5.8 Rates of Convergence
In this section we discuss some results that give the rate of convergence of M -estimators.
These results are useful as interrnediate steps in deriving alimit distribution, but also of
interest on their own. Applications include both classical estimators of "regular" parameters
and estimators that converge at a slower than ,Jn-rate. The main result is simple enough,
but its conditions include a maximal inequality, for which results such as in Chapter 19 are
needed.
Let JtD n be the empirical distribution of arandom sample of si ze n from a distribution
P, and, for every () in a metric space e, let X me (x) be a measurable function. Let en
(nearly) maximize the criterion function () ItD nme.
The criterion function may be viewed as the sum of the deterrninistic map () Pme
and the random fluctations f) ItDnme - Pme. The rate of convergence of en depends on
the combined behavior ofthese maps. Ifthe deterrninistic map changes rapidly as f) moves
away from the point of maximum and the random fluctuations are small, then en has a high
rate of convergence. For convenience of notation we measure the fluctuations in terrns of
the empirical process Gnme == ,Jn(ItDnme - Pme).
5.52 Theorem (Rate of convergence). Assume that for fixed constants C and a > fJ, for
every n, and for every sufficiently small 8 > O,
sup P (me - meo) < - C 8 a ,
d (e , eo) < 8
E * sup I G n (me - meo) I < C 8 {3 .
d (e , e o ) < 8
If the sequence en satisfies ItD nmen > JtD nmeo - o p (n a /(2{3-2a)) and converges in outer
probability to f)o, then n 1 /(2a-2{3)d(G n , f)o) == 0;(1).
76
M - and Z-Estimators
Proof. Set r n == n 1 /(2a-2{3) and suppose that en maximizes the map 8 1---+ IfDnme up to a
variable Rn == Op(r;;a).
For each n, the parameter space minus the point 8 0 can be partitioned into the "shells"
Sj,n == {8: 2 j - 1 < r n d(8,8 0 ) < 2 j }, with j ranging over the integers. If r n d(e n , 8 0 ) is
larger than 2 M for a given integer M, then en is in one of the shells Sj,n with j > M. In
that case the supremum of the map 8 1---+ IfD nme - IfD nmeo over this shell is at least - Rn by
the property of en. Conclude that, for every 8 > O,
p*(r n d(e n , 8 0 ) > 2M) < L p* ( sup (IP'nme - IP'nme o ) > - )
j?:.M eESj,n r n
2 j --:::8r n
+ P*(2d(e n , 8 0 ) > 8) + P(r Rn > K).
If the sequence en is consistent for 8 0 , then the second probability on the right converges
to O as n 00, for every fixed 8 > O. The third probability on the right can be made
arbitrarily small by choice of K, uniformly in n. Choose 8 > O small enough to ensure that
the conditions of the theorem hold for every 8 < 8. Then for every i involved in the sum,
we have
2(j-l)a
sup P(me - meo) < -c
eESj,n r
For C2(M-l)a > K, the series can be bounded in terms of the empirical process G n by
( 2(j-l)a ) (2i jrn)fJ 2r a
L P* IIGn(me - meo) Ilsj,n > c,J1i. 2 a < L ,J1i.2(j-l J a n ,
'>M r n ">M n
J- J-
2 j --:::8r n
by Markov's inequality and the definition of r n . The right side converges to zero for every
M == M n 00. .
Consider the special case that the parameter 8 is a Euclidean vector. If the map f) 1---+ P me
is twice-differentiable at the point of maximum 8 0 , then its first derivative at e a vanishes
and a Taylor expansion of the limit criterion function takes the form
P(me - meo) == (8 - 8 a )TV(8 - 8 0 ) + 0(118 - 8 0 11 2 ).
Then the first condition of the theorem holds with a == 2 provided that the second-derivative
matrix V is nonsingular.
The second condition of the theorem is a maximal inequality and is harder to verify. In
"regular" cases it is valid with f3 == 1 and the theorem yields the "usual" rate of convergence
-Jii. The theorem also applies to nonstandard situations and yields, for instance, the rate
n 1 / 3 if a == 2 and f3 == . Lemmas 19.34, 19.36 and 19.38 and corollary 19.35 are examples
of maximal inequalities that can be appropriate for the present purpose. They give bounds
in terms of the entropies of the classes of functions {me - me o : d (8, ( 0 ) < 8}.
A Lipschitz condition on the maps 8 1---+ me is one possibility to obtain simple estimates
on these entropies and is applicable in many applications. The result of the following
corollary is used earlier in this chapter.
5.8 Rates of Convergence
77
5.53 Corollary. For each e in an open subset of Euclidean space let x f--* me (x) be a
measurable function such that, for every el and e 2 in a neighborhood ofe a and a measurable
function in such that Pin 2 < 00,
Ime I (x) - me 2 (x)1 < m(x) IIe 1 - e 2 11.
Furthermore, suppose that the map e f--* Pme admits a second-order Taylor expansion at
the point of maximum e a with nonsingular second derivative. If ItD nmen > ItD nme o - O p (n -1 ),
p
then -Jfi(e n - €lo) == O p (1), provided that en ---+ e a .
Proof. By assumptian, the first condition of Theorem 5.52 is valid with ex == 2. To see
that the second ane is valid with f3 == 1, we apply Corollary 19.35 to the class of functions
:F == {me - meo: Ile - eoll < 8}. This class has enve lope function F == m 8, whence
l llml'P,20
E* sup IG n (mo - moo) I.:S / log N[] (E, F, L 2 (P)) dE.
lIe-eo II <o a
The bracketing entropy of the class :F is estimated in Example 19.7. Inserting the upper
bound obtained there in to the integral, we obtain that the preceding display is bounded
above by a multiple of
l llmllP,20 Fill 8 )
log - ds.
a s
Change the variables in the integral to see that this is a multiple of 8. .
Rates of convergence different from ,Jn are quite common for M -estimators of infinite-
dimensional parameters and may also be obtained through the application of Theorem 5.52.
See Chapters 24 and 25 for examples. Rates slower than ,Jn may also arise for fairly simple
parametric estimates.
5.54 Example (Moda I interval). Suppose that we define an estimator en oflocation as the
center of an interval of length 2 that contains the largest possible fraction of the observations.
This is an M-estimator for the functions me == 1[e-1,e+1].
For many underlying distributions the first condition of Theorem 5.52 holds with ex == 2.
It suffices that the map e f--* Pme == p[e - 1, e + 1] is twice-differentiable and has
a proper maximum at same point ea. Using the maximal inequality Corollary 19.35 (or
Lemma 19.38), we can show that the second condition is valid with f3 == . Indeed, the
bracketing entropy of the interval s in the real line is of the order 8/ S2, and the envelope
function of the class of functions 1[e-1,e+1] - 1[eo-1,e o +1] as e ranges over (e a - 8, e a + 8)
is baunded by 1[eo-1-0,00-1+0] + 1[eo+1-0,eo+1+0], whose squared L 2 -norm is bounded by
II p II 00 28.
Thus Theorem 5.52 applies with ex == 2 and f3 == and yields the rate of convergence
n 1/3. The resulting location estimator is very robust against outliers. However, in view of
its slow convergence rate, one should have good reasons to use il.
The use of an interval of length 2 is somewhat awkward. Every other fixed length would
give the same resull. More interestingly, we can also replace the fixed-length interval by the
smallest interval that contains a fixed fraction, for instance 1/2, of the observations. This
78
M - and Z -Estimatars
still yields a rate of convergence of nI /3. The intuitive reason for this is that the length of a
"shorth" settles down by a ,Jn-rate and hence its randomness is asymptotically negligible
relative to its center. D
The preceding theorem requires the consistency of en as a condition. This consistency is
implied if the other conditions are valid for every 8 > O, not just for small values of 8. This
can be seen from the proof or the more general theorem in the next section. Because the
conditions are not natural for large values of 8, it is usually better to argue the consistency
by other means.
5.8.1 Nuisance Parameters
In Chapter 25 we need an extension of Theorem 5.52 that allows for a "smoothing" or
"nuisance" parameter. We also take the opportunity to insert a number of other refinements,
which are sometimes useful.
Let x r-+ me, 1J (x) be measurable functions indexed by parameters «(), 17), and consider
estimators en contained in a set 8n that, for a given n contained in a set Hn, maximize the
map
() r-+ IP>nme,ryn'
The sets 8n and Hn need not be metric spaces, but instead we measure the discrepancies
between en and ()O, and n and alimiting value 170, by nonnegative functions () r-+ d1J «(), ()O)
and 17 r-+ d(17, 170), which may be arbitrary.
5.55 Theorem. Assume that, for arbitrary functions en: 8n x Hn r-+ JR and 4Jn: (O, (0) r-+
JR such that 8 r-+ 4Jn (8) /8 f3 is decreasing for some tJ < 2, every «(), 17) E 8n x Hn, and
every 8 > O,
P(me,1J - meO,1J) + e n «(), 17) < -d;«(), ()O) + d 2 (17, 170),
E* sup IG n (me,1J - meO,1J) - ,Jn e n «(), 17)1 < 4Jn(8).
dT} «(},(}o)<8
«(},T})E8 n xH n
Let 8n > O satisfy 4Jn(8 n ) < ,Jn 8 for every n. If p(e n E 8n, n E Hn) -+ 1 and
IP>nmen,ryn > IP>nmeO,ryn - Op(8 ), thendryn(en,()o) == O;(8n +d( n, 170)).
Proof. For simplicity assume that IP>nmen,ryn > IP>nmeO,ryn' without a tolerance term. For
each n E N, j E Z and M > O, let Sn,J,M be the set
{c(), 17) E 8n x H n :2 J - 1 8 n < d1J«()'()o) < 2 J 8 n ,dC17, 170) < 2- M d 1J «(),()0)}.
Thentheintersectionoftheeventsce n , n) E 8nxHn,anddryncen, ()o) > 2M(8n+dC n' 170))
is contained in the union of the events {(en, n) E Sn,J,M} over j > M. By the definition
of en, the supremum of IP>n(me,1J - meO,1J) over the set of parameters «(), 17) E Sn,J,M is
nonnegative on the event {(en, n) E Sn,J,M}. Conclude that
P* (d n (en, e o ) > 2 M (on + d(Tln, 170))' (en, Tin) E en X Hn)
< L P* ( SUP IP> n C m e,1J - meO,1J) > O ) .
J M (e,1J)ES n .j.M
5.9 Argmax Theorem
79
For every j, ((), 17) E Sn,j,M, and every sufficiently large M,
P(me,TJ - m(jo,TJ) + e n ((), 17) < -d;((), ( 0 ) + d 2 (17, 170)
< _ ( 1 - 2-2M ) d 2 ( 8 8 ) < _2 2j - 4 8 2 .
- TJ ' o - n
From here on the proof is the same as the proof of Theorem 5.52, except that we use that
CPn (co) < c f3 cpn (8) for every c > 1, by the assumption on CPn. .
*5.9 Argmax Theorem
The consistency of a sequence of M -estimators can be understood as the points of maximum
en of the criterion functions () r-+ M n (()) converging in probability to a point of maximum
of the limit criterion function () r-+ M(8). So far we have made no attempt to understand
the distributionallimit properties of a sequence of M -estimators in a similar way. This is
possible, but it is somewhat more complicated and is perhaps best studied after developing
the theory of weak convergence of stochastie processes, as in Chapters 18 and 19.
Because the estimators en typically converge to constants, it is necessary to rescale them
before studying distributionallimit properties. Thus, we start by searching for a sequence
ofnumbers r n 1-+ 00 such that the sequence h n == rn(e n - 8) is uniformly tight. The resuits
of the preceding section should be useful. If en maximizes the function 8 r-+ M n (e), then
the rescaled estimators h n are maximizers of the local criterion functions
h f-+ M n (e + ) - Mn(e o ).
Suppose that these, if suitably normed, converge to alimit process h r-+ M (h). Then the
general principle is that the sequence h n converges in distribution to the maximizer of this
limit process.
For simplicity of notation we shalI write the loeal eriterion funetions as h r-+ M n (h).
Let {Mn (h): h E Hn} be arbitrary stochastic processes indexed by subsets Hn of a given
metric space. We wish to prove that the argmax-functional is eontinuous: If M n -"M M and
Hn -+ Hin a suitable sense, then the (ne ar) maximizers h n of the randommaps h r-+ M n (h)
converge in distribution to the maximizer h of the limit process h r-+ M (h). It is easy to
find examples in which this is not true, but given the right definitions it is, under some
eonditions. Given a set B, set
M(B) == supM(h).
hEB
Then convergenee in distribution of the vectors (Mn (A), M n (B)) for given pairs of sets A
and B is an appropriate form of convergence of M n to M. The following theorem gives some
flexibility in the ehoice of the index ing sets. We implicitly either assume that the suprema
M n (B) are measurable or understand the weak convergence in term s of outer probabilities,
as in Chapter 18.
The result we are looking for is not likely to be true if the maximizer of the limit proeess
is not well defined. Exaetl y as in Theorem 5.7, the maximum should be "well separated."
Because in the present case the limit is a stochastic process, we require that every sample
path h 1-+ M(h) possesses a well-separated maximum (condition (5.57)).
80
M - and Z-Estimators
5.56 Theorem (Argmax theorem). Let M n and M be stochastic processes indexed by sub-
sets Hn and H of a given metric space such that, for every pair of a closed set F and a set
K in a given collection lC,
(Mn(F n K n H n ), Mn(K n Hn)) (M(F n K n H), M(K n H)).
Furthermore, suppose that every sample path of the process h 1---+ M (h) possesses a well-
separated point of maximum h in that, for every open set G and every K E JC,
M(h) > M(G c n K n H),
if h E G,
a.s..
(5.57)
IfMn(h n ) > Mn(Hn)-op(l)andforevery£ > OthereexistsK E lCsuchthatsuPnP(hn
K) < £ and P(h K) < £, then h n h.
Proof. If h n E F n K, then Mn(F n K n Hn) > Mn(B) - opel) for any set B. Renee,
for every closed set F and every K E lC,
P(h n E F n K) < P(Mn(F n K n Hn) > Mn(K n Hn) - opel))
< P(M(F n K n H) > M(K n H)) + 0(1),
by Slutsky's lemma and the portmanteau lemma. If h E F C , then M(F n K n H) is strictly
smaller than M(h) by (5.57) and hence on the intersection with the event in the far right
side h cannot be contained in K n H. It follows that
limsupP(h n E F n K) < P(h E F) + P(h ti- K n H).
By assumption we can choose K such that the left and right sides change by less than £ if
we replace K by the whole space. Renee h n h by the portmanteau lemma. .
The theorem works most smoothly if we can take JC to consist only of the whole space.
Rowever, then we are close to assuming some sort of global uniform convergence of M n
to M, and this may not hold or be hard to proveo It is usually more economical in terms
of conditions to show that the maximizers h n are contained in certain sets K, with high
probability. Then uniform convergence of M n to M on K is sufficient. The choice of
compact sets K corresponds to establishing the uniform tightness of the sequence h n before
applying the argmax theorem.
Ifthe sample paths of the processes M n are bounded on K and Hn == H for every n, then
the weak convergence of the processes M n viewed as elements of the space foo(K) implies
the convergence condition of the argmax theorem. This follows by the continuous-mapping
theorem, because the map
Z 1---+ (z(A nK), zeB n K))
from foo (K) to JR2 is continuous, for every pair of sets A and B. The weak convergence in
foo(K) remain s sufficient if the sets Hn depend on n but converge in a suitable way. Write
Hn H if H is the set of alllimits lim h n of converging sequences h n with h n E Hn
for every n and, more over, the limit h == limi h ni of every converging sequence h ni with
h ni E H ni for every i is contained in H.
5.9 Argmax Theorem
81
5.58 Corollary. Suppase that M n M in £00 (K) for every compact subset K of JR.k, for
alimit process M with continuous sample paths that have unique points of maxima h. If
Hn ---+ H, M n (h n ) > M n (Hn) - o p (1), and the sequence h n is uniformly tight, then h n h.
Proof The compactness of K and the continuity of the sample paths h f---+ M (h) imply
that the (unique) points of maximum h are automatically well separated in the sense of
(5.57). Indeed, if this fails for a given open set G :3 h and K (and a given w in the
underlying probability space), then there exists a sequence h m in G C n K n H such that
M (h m ) ---+ M (h). If K is compact, then this sequence can be chosen convergent. The limit
ho must be in the closed set G C and hence cannot be h. By the continuity of M it also has
the property that M(h o ) == limM(h m ) == M(h). This contradicts the assumption that h is
a unique point of maximum.
If we can show that (Mn (F n H n ), M n (K n Hn)) converges to the corresponding limit
for every compact sets F cK, then the theorem is acorollary of Theorem 5.56. If Hn == H
for every n, then this convergence is immediate from the weak convergence of M n to M
in £oo(K), by the continuous-mapping theorem. For Hn changing with n this convergence
may fail, and we ne ed to refine the proof of Theorem 5.56. This goe s through with minor
changes if
limsupP(Mn(F n Hn) - Mn(K ( I Hn) > x) < P(M(F n H) - M(K n H) > x),
n ---+ 00
for every x, every compact set F and every large closed ball K. Define functions gn: £00 (K)
r-+ JR. by
gn(Z) == sup z(h) - sup z(h),
hEFnH n hEKnH n
and g similarly, but with H replacing Hn. By an argument as in the proof of Theo-
rem 18.11, the desired result follows if lim sup gn (Zn) < g(z) for every sequence Zn ---+ Z
in £oo(K) and continuous function z. (Then lim sup P(gn(M n ) > x) < P(g(M) > x) for
every x, for any wealdy converging sequence M n M with a limit with continuous sample
paths.) This in turn follows if for every precompact set B CK,
sup z(h) < lim sup Zn (h) < sup z(h).
hEBnH n---+oo hEBnH I1 hEBnH
To prove the upper inequality, select h n E B n Hn such that
sup zn(h) == zn(h n ) + 0(1) == z(h n ) + 0(1).
h EBnH n
Because B is compact, every subsequence of h n has a converging subsequence. Because
Hn ---+ H, the limit h must be in B n H. Because z(h n ) ---+ z(h), the upper bound follows.
To prove the lower inequality, select for given 8 > O an element h E f3 n H such that
sup z(h) < z(h) + 8.
hEBnH
Because Hn ---+ H, there exists h n E Hn with h n ---+ h. This sequence must be in f3 c B
eventually, whence z(h) == lim z(h n ) == lim zn (h n ) is bounded above by lim inf sUPhEBnH n
zn(h). .
82
M- and Z-Estimators
The argmax theorem can also be used to prove consistency, by applying it to the original
criterion functions e f--+ M n (e). Then the limit process e f--+ M (e) is degenerate, and has
a fixed point of maximum e o . Weak convergence becomes convergence in probability, and
the theorem now gives conditions for the consistency en e o . Condition (5.57) reduces
to the well-separation of e o , and the convergence
p
sup M n (e) sup M n (e)
8EFnKn8 n 8EFnKn8
is, apart from allowing en to depend on n, weaker than the uniform convergence of M n to
M.
Notes
In the section on consistency we have given two main results (uniform convergence and
Wald's proof) that have proven their value over the years, but there is more to say on this
subject. The two approaches can be unified by replacing the uniform convergence by "one-
sided uniform convergence," which in the case of i.i.d. observations can be established
under the conditions of Wald's theorem by a bracketing approach as in Example 19.8 (but
then one-sided). Furthermore, the use of special properties, such as convexity of the ljf or
m functions, is often helpful. Examples such as Lemma 5.10, or the treatment of maximum
likelihood estimators in exponential families in Chapter 4, appear to indicate that no single
approach can be satisfactory.
The study of the asymptotic properties of maximum likelihood estimators and other
M -estimators has a long history. Fisher [48], [50] was a strong advocate of the method of
maximum likelihood and noted its asymptotic optimality as early as the 1920s. What we
have labelled the classical conditions correspond to the rigorous treatment given by Cramer
[27] in his authoritative book. Huber initiated the systematic study of M -estimators, with
the purpose of developing robu st statistical procedures. His paper [78] contains important
ideas that are precursors for the application of techniques from the theory of empirical
processes by, among others, Pollard, as in [117], [118], and [120]. For one-dimensional
parameters these empirical process methods can be avoided by using a maximal inequality
based on the L 2 -norm (see, e.g., Theorem 2.2.4 in [146]). S urprisingly, then a Lipschitz
condition on the Hellinger distance (an integrated quantity) suffices; see for example, [80] or
[94]. For higher-dimensional parameters the results are also not the best possible, but I do
not know of any simple better ones.
The books by Huber [79] and by Hampel, Ronchetti, Rousseeuw, and Stahel [73] are
good sources for applications of M -estimators in robust statisties. These reference s also
discuss the relative efficiency of the different M -estimators, which motivates, for instance,
the use of Huber' s ljf - function. In this chapter we have derived Huber' s estimator as the
solution of the problem of minimizing the asymptotic variance under the side condition
of a uniformly bounded influence function. Originally Huber derived it as the solution to
the problem of minirnizing the maximum asymptotic variance sup p (J for P ranging over
a contamination neighborhood:P == (1 - 8)cp + 8Q with Q arbitrary. For M-estimators
these two approaches turn out to be equivalent.
The one-step method can be traced back to numerical schemes for solving the likelihood
equations, including Fisher's method of scoring. One-step estimators were introduced for
Problems
83
their asymptotic efficiency by Le Cam in 1956, who later developed them for generallocally
asymptotically quadratic models, and also introduced the discretization device, (see [93]).
PROBLEMS
1. Let XI, . . . , X n be a sample from a density that is strict1y positive and symmetric about some
point. Show that the Huber M -estimator for location is consistent for the symmetry point.
2. Find an expression for the asymptotic variance of the Huber estimator for location if the obser-
vations are normally distributed.
3. Define 'ljJ(x) == 1 - p, O, p if x < O, O, > O. Show that E1/J(X - e) == O implies that P(X <
e) p P(X e).
4. Let XI, . . . , X n be i.i.d. N(f.,i, ( 2 )-distributed. Derive the maximum likelihood estimator for
(f.,i, 0'2) and show that it is asymptotically norma!. Calculate the Fisher information matrix for
this parameter and its inverse.
5. Let XI, . . . , X n be i.i.d. Poisson(lje)-distributed. Derive the maximum likelihood estimator for
e and show that it is asymptotically norma!.
6. Let XI, . . . , X n be i.i.d. N (e, e)-distributed. Derive the maximum likelihood estimator for e
and show that it is asymptotically norma!.
7. Find a sequence of fixed (nonrandom) functions M n : JR. r---+ JR. that converges pointwise to alimit
Mo and such that each M n has a unique maximum at a point en, but the sequence en does not
converge to eo. Can you also find a sequence M n that converges uniformly?
8. Find a sequence of fixed (nonrandom) functions M n : JR. r---+ JR. that converges pointwise but not
uniformly to a limit Mo such that each M n has a unique maximum at a point en and the sequence
en converges to eo.
9. Let XI, . . . , X n be i.i.d. observations from a uniform distribution on [O, eJ. Show that the
sequence of maximum likelihood estimators is asymptotically consistent. Show that it is not
asymptotically norma!.
10. Let XI, . . . , X n be i.i.d. observations from an exponential density e exp( -ex). Show that the
sequence of maximum likelihood estimators is asymptotically norma!.
11. Let IF; 1 (p) be a pth sample quantile of a sample from a cumulative distribution P on IR. that is
differentiable with positive derivative at the population pth-quantile F- 1 (p) == inf{ x: F (x) >
p }. Show that ,jn (JF; 1 (p) - p-I (p)) is asymptotically normal with mean zero and variance
p (1 - p) j f ( F -1 (p) ) 2 .
12. Derive a minimal condition on the distribution function F that guarantees the consistency of the
sample pth quantile.
13. Calculate the asymptotic variance of ,jn(e n - e) in Example 5.26.
14. Suppose that we observe arandom sample from the distribution of (X, Y) in the following
errors-in- variables model:
X==Z+e
Y Ž == ex + f3 Z + f,
where (e, f) is bivariate normally distributed with mean O and covariance matrix 0'2 I and is
independent from the unobservable variable Z. In analogy to Example 5.26, construct a system
of estimating equations for (ex, f3) based on a conditionallikelihood, and study the limit properties
of the corresponding estimators.
15. In Example 5.27, for what point is the least squares estimator en consistent if we drop the
condition that E(e I X) == O? Derive an (implicit) solution in terms of the function E(e I X). Is
it necessarily eo if Ee == O?
84 M - and Z-Estimators
16. In Example 5.27, consider the asymptotic behavior of the least absolute-value estimator () that
minimizes L7=II Yi - CPe (Xi) I.
17. LetXI, ..., X n bei.i.d. withdensity fA,a(x) == Ae- A (x-a)l{x > a}, wheretheparametersA > O
and a E IR are unknown. Calculate the maximum likelihood estimator ().n, an) of (A, a) and
derive its asymptotic properties.
18. Let X be Poisson-distributed with density pe (x) == ex e-e / X!. Show by direct calculation that
Eele (X) == O and Eele (X) == - Eel (X). Compare this with the assertions in the introduction.
Apparently, differentiation under the integral (sum) is permitted in this case. Is that obvious from
results from measure theory or (complex) analysis?
19. Let XI, . . . , X n be a sample from the N (e, 1) distribution, where it is known that e > O. Show
that the maximum likelihood estimator is not asymptotically normal under e == O. Why does this
not contradict the theorems of this chapter?
20. Show that (e - eo)\I1 n (en) in formula (5.18) converges in probability to zero if en eo, and that
there exists an integrable function M and 8 > O with 11f e (x) I < M (x) for every x and every
II e - eo II < 8.
21. If en maximizes M n , then it also maximizes M:. Show that this may be used to relax the
conditions of Theorem 5.7 to sUPe IM: - M+I(e) -+ Oinprobability (if M(eo) > O).
22. Suppose that for every £ > O there exists a set B € with lim inf P (e nEe €) > 1 - £. Then uniform
convergence of M n to M in Theorem 5.7 can be relaxed to uniform convergence on every B€.
23. Show that Wald's consistency proofyields almost sure convergence of () n, rather than convergence
in probabili ty if the parameter space is compact and M n (e n) > M n (eo) - O (1) .
24. Suppose that (XI, YI), . . . , (X n , Y n ) are i.i.d. and satisfy the linear regression relationship Yi ==
eT Xi + ei for (unobservable) errors el, . . . , en independent of XI, . . . , Xn. Show that the mean
absolute deviation estimator, which minimizes L I Yi - ex i I, is asymptotically normal under a
mild condition on the error distribution.
25. (i) Verify the conditions of Wald' s theorem for me the log likelihood function of the N (/1-, (J'2)_
distribution if the parameter set for e == (/1-, (J' 2 ) is a compact subset of JR x JR + .
(ii) Extend me by continuity to the compactification of JR x JR+. Show that the conditions of
Wald's theorem fail at the points (/1-, O).
(iii) Replace me by the log likelihood function of a pair of two independent observations from the
N (/1-, (J'2)-distribution. Show that Wald's theorem now does apply, also with a compactified
parameter set.
26. A distribution on JRk is called ellipsoidally symmetric if it has a density of the form x 1---+
g((x -/1-)T -I(x -/1-)) for a function g:[O,oo) 1---+ [0,(0), a vec tor /1-, and a symmetric
positive-definite matrix . Study the Z -estimators for location {l that solve an equation of the
form
n
" ( T"-I )
L 1/1 (Xi - /1-) n (Xi - /1-) ,
i=I
for given estimators n and, for instance, Huber's 1/1-function. Is the asymptotic distribution of
n important?
27. Suppose that B is a compact metric space and M: B -+ IR is continuous. Show that (5.8) is
equivalent to the point eo being a point of unique global maximum. Can you relax the continuity
of M to some form of "semi -continuity"?
6
Contiguity
"Contiguity" is another name for "asymptotic absolute continuity."
Contiguity arguments are a technique to obtain the limit distribution
of a sequence of statistics under underlying laws Qn from alimiting
distribution under laws Pn. Typically, the laws Pn describe a null distri-
bution under investigation, and the laws Qn correspond to an alternative
hypothesis.
6.1 Likelihood Ratios
Let P and Q be measures on a measurable space (Q, A). Then Q is absolutely continuous
with respect to P if P (A) == O implies Q (A) == O for every measurable set A; this is denoted
by Q « P. Furthermore, P and Q are orthogonal if Q can be partitioned as Q == Qp U QQ
with Qp n QQ == 0 and P(QQ) == O == Q(Qp). Thus P "charges" only Qp and Q "lives
on" the set Q Q, which is disjoint with the "support" of P. Orthogonality is denoted by
p 1- Q.
In general, two measures P and Q need be neither absolutely continuous nor orthogonal.
The relationship between their supports can best be described in term s of densities. Suppose
P and Q possess densities p and q with respect to a measure J-L, and consider the sets
Qp == {p > O},
QQ == {q > O}.
See Figure 6.1. Because P(Q ) == fp=o p dJ-L == O, the measure P is supported on the set
Qp. Similarly, Q is supported on QQ. The intersection Qp n QQ receives positive measure
from both P and Q provided its measure under J-L is positive. The measure Q can be written
as the sum Q == Qa + Ql. of the measures
Qa(A) == Q(A n {p > O});
Ql.(A) == Q(A n {p == O}).
(6.1)
As proved in the next lemma, Qa « p and Ql. 1- P. Furthermore, for every measurable
set A
Qa(A) == [ q dP.
JA p
The decomposition Q == Qa + Ql. is called the Lebesgue decomposition of Q with respect
to P. The measures Qa and Ql. are called the absolutely continuous part and the orthogonal
85
86
Contiguity
Q
p>O
q=O
p=O
q>O
p=q=O
Figure 6.1. Supports of measures.
part (ar singular part) of Q with respect to P, respectively. In view of the preceding display,
the function q / p is a density of Qa with respect to P. It is denoted d Q/ d P (not: d Qa / d P),
so that
dQ q
dP p
p - a.s.
As long as we are only interested in the properties of the quotient q / p under P -probability,
we may leave the quotient undefined for p == O. The density d Q / d P is only P -almost
surely unique by definition. Even though we have used densities to define them, d Q / d P
and the Lebesgue decomposition are actuaHy independent of the choice of densities and
dorninating measure.
In statistics a more common name for a Radon-Nikodym density is likelihood ratio. We
shaH think of it as arandom variable d Q / d P : Q r-+ [O, (0) and shaH study its law under P.
6.2 Lemma. Let P and Q be probability measures with densities p and q with respect to
a measure 11. Thenfor the measures Qa and Q..l defined in (22.30)
(i) Q == Qa + Q..l, Qa « P, Q..l 1- P.
(ii) Qa(A) == JA (q/p) dP for every measurable set A.
(iii) Q« P if and only if Q(p == O) == O if and only if J (q / p) d P == 1.
Proof. The first statement of (i) is obvious from the definitions of Qa and Q..l. For the
second, we note that P (A) can be zero only if p(x) == O for l1-almost all X E A. In this
case, I1(A n {p > O}) == O, whence Qa(A) == Q(A n {p > O}) == O by the absolute
continuity of Q with respect to 11. The third statement of (i) follows from P (p == O) == O
and Q..l (p > O) == Q (0) == O.
Statement (ii) follows from
1 1 q l q
Qa(A) == q dl1 == - P dl1 == - dP.
An{p>O} An{p>O} P A P
For (iii) we note first that Q « p if and only if Q..l == O. By (22.30) the latter happens if
and only if Q (p == O) == O. This yields the first "if and only if." For the second, we note
6.2 Contiguity
87
that by (ii) the total mass of Qa is equal to Qa (Q) == J (q / p) d P. This is 1 if and only if
Qa == Q. .
It is not true in general that J f d Q == J f (d Q / d P) d P. For this to be true for every
measurable function f, the measure Q must be absolutely continuous with respect to P.
On the other hand, for any P and Q and nonnegative f,
j fdQ > r fqdfL= r f q PdfL= j f dQ dP,
Jp>o Jp>o p dP
This inequality is used freely in the following. The inequality may be strict, because dividing
by zero is not permitted. t
6.2 Contiguity
If a probability measure Q is absolutely continuous with respect to a probability measure
P, then the Q -law of arandom vector X : Q 1---+ IR k can be calculated from the P -law of the
pair (X, dQ/dP) through the formula
dQ
EQf(X) == Epf(X) dP .
With pX,v equal to the law of the pair (X, V) == (X, dQ/dP) under P, this relationship
can also be expressed as
dQ J
Q(X E B) == E p l B (X)- == v dpx,v (x, v).
d P BxIR
The validity of these formulas depends essentially on the absolute continuity of Q with
respect to P, because a part of Q that is orthogonal with respect to P cannot be recovered
from any P-law.
Consider an asymptotic version of the problem. Let (Qn, An) be measurable spaces,
each equipped with a pair of probability measures Pn and Qn. Under what conditions can
a Qn-limit law of random vectors X n : Qn 1---+ IR k be obtained from suitable Pn-limit laws?
In view of the above it is necessary that Qn is "asymptotically absolutely continuous" with
respect to Pn in a suitable sense. The right concept is contiguity.
6.3 Definition. The sequence Qn is contiguous with respect to the sequence Pn if
Pn (An) -+ O implies Qn (An) -+ O for every sequence of measurable sets An. This is
denoted Qn <J Pn. The sequences Pn and Qn are mutually contiguous ifboth Pn <J Qn and
Qn <J Pn. This is denoted Pn <J !>Qn.
The name "contiguous" is standard, but perhaps conveys a wrong image. "Contiguity"
suggests sequences of probability measures living next to each other, but the correct image
is "on top of each other" (in the limit).
t The algebraic identify dQ = (dQ/dP) dP is false, beeause the notation dQ/dP is used as shorthand for
d Qa / d P: If we write d Q / d P, then we are not implieit1y assuming that Q « P.
88
Contiguity
Before answering the question of interest, we give two characterizations of contiguity
in terms of the asymptotic behavior of the likelihood ratios of Pn and Qn. The likelihood
ratios d Qn/ d Pn and d Pn/ d Qn are nonnegative and satisfy
dQn
E P - < 1
n dPn -
dPn
and E Q - < 1.
n d Qn -
Thus, the sequences of likelihood ratios d Qn/ d Pn and d Pn/ d Qn are uniformly tight under
Pn and Qn, respectively. By Prohorov's theorem, every subsequence has a further weak1y
converging subsequence. The next lemma shows that the properties of the limit points
determine contiguity. This can be understood in analogy with the nonasymptotic situation.
For probability measures P and Q, the following three statements are equivalent by (iii) of
Lemma 6.2:
Q « P,
Q( =O)=O,
dQ
E p - == 1.
dP
This equivalence persists if the three statements are replaced by their asymptotic counter-
parts: Sequences Pn and Qn satisfy Qn <] Pn, if and only ifthe weak limit points of d Pn/ d Qn
under Qn give mass O to O, if and only if the weak limit points of d Qn / d Pn under Pn have
mean 1.
6.4 Lemma (Le Cam's first [emma). Let Pn and Qn be sequences ofprobability measures
on measurable spaces (On, An). Then the following statements are equivalent:
(i) Qn <] Pn.
(ii) If dPn/d Qn & U along a subsequence, then P(U > O) == 1.
(iii) If d Qn/dPn V along a subsequence, then EV == 1.
(iv) For any statistics Tn : On ffi.k: If Tn !+ O, then Tn O.
Proof. The equivalence of (i) and (iv) follows directly from the definition of contiguity:
Given statistics Tn, consider the sets An == {II Tn II > £}; given sets An, consider the statistics
Tn == 1 An .
(i) ==> (ii). For simplicity of notation, we write just {n} for the given subsequence
along which d Pn/ d Qn & U. For given n, we define the function gn (£) == Qn (d Pn/ d Qn <
£) - P(U < £). By the portmanteau lemma, liminf gn(£) > O for every £ > O. Then, for
£n + O at a sufficiently slow rate, also lim inf gn (£n) > O. Thus,
P(U = O) = limP(U < IOn) < liminfQn( : : < IOn).
On the other hand,
( dPn ) 1 dPn f
Pn d < £n /\ qn > O == d d Qn < £n d Qn -+ O.
Qn dPn/dQn 8n Qn
If Qn is contiguous with respect to Pn, then the Qn-probability of the set on the left goes
to zero also. But this is the probability on the right in the first display. Combination shows
that P(U == O) == O.
(iii) ==> (i). If Pn (An) -+ O, then the sequence 1r2 n - A n converges to 1 in Pn-probability.
By Prohorov's theorem, every subsequence of {n} has a further subsequence along which
6.2 Contiguity
89
(dQn/dPn, 1r2 n - A n) -v-7 (V, 1) under Pn, for some weak limit V. The function (u, t) ut
is continuous and nonnegative on the set [O, (0) X {O, 1}. By the portmanteau lemma
liminfQn(Qn - An) > liminf f Ir/n-An ;: dPn > El. V.
Under (iii) the right side equals EV == 1. Then the left side is 1 as well and the sequence
Qn (An) == 1 - Qn (S1 n - An) converges to zero.
(ii) =} (iii). The probability measures J-Ln == (Pn + Qn) dominate both Pn and Qn, for
every n. The sum of the densities of Pn and Qn with respect to ILn equals 2. Rence, each of
the densities takes its values in the compact interval [0,2]. By Prohorov's theorem every
subsequence possesses a further subsequence along which
dPn Qn d Qn Pn dPn Rn
- -v-7 U, - -v-7 V, W n :== - -v-7 W,
dQn dPn dILn
for certain random variables U, V and W. Every W n has expectation 1 under J-Ln' In view
of the boundedness, the weak convergence of the sequence W n implies convergence of
moments, and the limit variable has mean E W == 1 as well. For a given bounded, continuous
functionf,defineafunctiong: [O, 2] ffi.by g(w) == f(w/(2-w))(2-w) for O < w < 2
andg(2) == O. Thengisboundedandcontinuous. BecausedPn/dQn == W n /(2- W n ) and
d Q n / d ILn == 2 - W n , the portmanteau lemma yields
( d Pn ) ( d Pn ) d Qn ( W )
EQJ dQn = EfLJ dQn df.Ln = EfLng(W n ) --+ Ef 2 _ W (2 - W),
where the integrand in the right side is understood to be g(2) == O if W == 2. Byassumption,
the left side converges to Ef(U). Thus Ef(U) equals the right side of the display for every
continuous and bounded function f. Take a sequence of such functions with 1 > f m {, 1 {O},
and conclude by the dominated-convergence theorem that
P(U == O) == E1{o} (U) == E1{o} ( W ) (2 - W) == 2P(W == O).
2-W
By a similar argument, Ef (V) == Ef (2 - W) / W) W for every continuous and bounded
function f, where the integrand on the right is understood to be zero if W == O. Take a
sequence O < fm (x) t x and conclude by the monotone convergence theorem that
( 2 - W )
EV == E W W == E(2 - W)l w >o == 2P(W > O) - 1.
Combination of the last two displays shows that P(U == O) + EV == 1. .
6.5 Example (Asymptotic log normality). The following special case plays an important
role in the asymptotic theory of smooth parametric models. Let Pn and Qn be probability
measures on arbitrary measurable spaces such that
d Pn & eNCIL,a2) .
dQn
Then Qn <J Pn. Furthermore, Qn <J [> Pn if and only if IL == - a2.
Because the (log norma!) variable on the right is positive, the first assertion is immediate
from (ii) of the theorem. The second follows from (iii) with the role s of Pn and Qn switched,
on noting that E exp N (/.-L, ( 2 ) == 1 if and only if IL == - a2.
90
Contiguity
A mean equal to minus half times the variance looks peculiar, but we shalI see that this sit-
uation arises naturalIy in the study of the asymptotic optimality of statistical procedures. O
The following theorem solves the problem of obtaining a Qn-limit law from aPn-limit
law that we posed in the introduction. The result, a version of Le Cam's third lemma, is in
perfect analogy with the nonasymptotic situation.
6.6 Theorem. Let Pn and Qn be sequences of probability measures on measurable spaces
(Qn, An), and let X n : Qn IR k be a sequence of random vectors. Suppose that Qn <J Pn
and
( x dQn ) ( X V ) .
n, d Pn '
Then L(B) == EI B (X) V defines a probability measure, and X n & L.
Proof. Because V > O, it folIows with the help of the monotone convergence theorem
that L defines a measure. By contiguity, EV == 1 and hence L is a probability measure.
It is immediate from the definition of L that J f dL == Ef(X) V for every measurable
indicator function f. Conclude, in steps, that the same is true for every simple function f,
any nonnegative measurable function, and every integrable function.
If f is continuous and nonnegative, then so is the function (x, v) f (x) v on JR.k x
[O, (0). Thus
liminf EQJ(Xn) > liminf f f(Xn) ;: dPn > Ef(X)V,
by the portmanteau lemma. Apply the portmanteau lemma in the converse direction to
conclude the proof that X n & L. .
6.7 Example (Le Cam's third lemma). The name Le Cam' s third [emma is often reserved
for the folIowing result. If
( d Qn ) Pn ( ( J.L )
X n , log dPn NHI -4 CT2 '
( :2))'
then
X n & Nk(J.L + T, b).
In this situation the asymptotic covariance matrices of the sequence X n are the same under
Pn and Qn, but the mean vectors differ by the asymptotic covariance Tbetween X n and the
log likelihood ratios. t
The statement is a special case of the preceding theorem. Let (X, W) have the given
(k + 1) -dimensional normal distribution. By the continuous mapping theorem, the sequence
(X n , d Qn/ d Pn) converges in distribution under Pn to (X, e w). Because W is N (- a2, a 2 )_
distributed, the sequences Pn and Qn are mutualIy contiguous. According to the abstract
t We set log O = -00; beeause the normal distribution does not eharge the point -00 the assumed asymptotie
normality oflogdQn/dPn includes the assumption that Pn(dQn/dPn = O) -+ O.
Problems
91
version of Le Cam's third lemma, X n & L with L(B) == E1 B (X)e w . The characteristic
function of L is J e i t T X dL (x) == Ee žtT X e w. This is the characteristic function of the given
normal distribution at the vector Ct, -i). Thus
žtTJ.l_1(J'2_1(tT -i) ( L T )( t )
f , t T 2 2' TT (J'2 -i ' t T ( +T) _l t T "" t
el X dL(x) == e == el J.l 2 L...
The right side is the characteristic function of the Nk ({L + T, IJ distribution. D
Notes
The concept and theory of contiguity was developed by Le Cam in [92]. In his paper the
results that were later to become known as Le Cam's lemmas are listed as asingle theorem.
The names "first" and "third" appear to originate from [71]. (The second lemma is on
product measures and the first lemma is actually only the implication (iii) ::::} (i).)
PROBLEMS
1. Let Pn == N (O, 1) and Qn == N (fJ-n, 1). Show that the sequences Pn and Qn are mutually
contiguous if and only if the sequence fJ-n is bounded.
2. Let P n and Q n be the distribution of the mean of a sample of size n from the N (O, 1) and the
N(()n, 1) distribution, respectively. Show that Pn <] t>Qn if and only if ()n == 0 (1/,J1i).
3. Let Pn and Qn be the lawofa sample ofsizen from the uniformdistribution on [O, 1] or [O, 1 + lin],
respectively. Show that Pn <] Qn. Is it also true that Qn <] Pn? Use Lemma 6.4 to derive your
answers.
4. Suppose that II Pn - Qn II -+ O, where II . II is the total variation distance II P - Q II == sUPA I P (A) -
Q(A) I. Show that Pn <] t> Qn.
5. Given 8 > O find an example of sequences such that Pn <] t> Qn, but IIPn - Qn II -+ 1 - 8. (The
maximum total variation distance between two probability measures is 1.) This exercise shows
that it is wrong to think of contiguous sequences as being close. (Try measures that are supported
on just two points.)
6. Give a simple example in which Pn <] Qn, but it is not true that Qn <] Pn.
7. Show that the constant sequences {P} and {Q} are contiguous if and only if P and Q are absolutely
continuous.
8. rf p « Q, then Q(An) -+ O implies P(An) -+ O for every sequence of measurable sets. How
does this follow from Lemma 6.4 ?
7
Local Asymptotic Normality
A sequenee of statistieal models is "loeally asymptotieally normai" if,
asymptotieally, their likelihood ratio proeesses are similar to those for
a normal loeation parameter. Teehnically, this is if the likelihood ratio
proeesses admit a eertain quadratie expansion. An important example
in whieh this arises is repeated sampling from a smooth parametrie
model. Loeal asymptotie normality implies eonvergence of the models to
a Gaussian model after a resealing of the parameter.
7.1 Introduction
Suppose we observe a sample XI, . . . , X n from a distribution P() on some measurable space
(X, A) indexed by a parameter g that ranges over an open subset 8 of IR k . Then the full
observation is asingle observation from the product P; of n copies of p(), and the statis-
tical model is completely described as the collection of probability measures {P; : g E 8}
on the sample space (X n , An). In the context of the present chapter we shall speak of a
statistieal experiment, rather than of a statistical model. In this chapter it is shown that
many statistical experiments can be approximated by Gaussian experiments after a suitable
reparametrization.
The reparametrization is centered around a fixed parameter go, which should be regarded
as known. We define a loeal parameter h == ,jn(g - go), rewrite P; as P():+h/vn' and thus
obtain an experiment with parameter h. In this chapter we show that, for large n, the
experiments
(P +h/.Jii : h E ]] n and ( N(h, Ie l) : h E JRk)
are similar in statistical properties, whenever the original experiments () 1---+ P() are "smooth"
in the parameter. The second experiment consists of observing asingle observation from a
normal distribution with mean h and known covariance matrix (equal to the inverse of the
Fisher information matrix). This is a simple experiment, which is easy to analyze, whence
the approximation yields much information about the asymptotic properties of the original
experiments. This information is extracted in several chapters to follow and concerns both
asymptotic optimality theory and the behavior of statistical procedures such as the maximum
likelihood estimator and the likelihood ratio test.
92
7.2 Expanding the Likelihood
93
We have taken the loe al parameter set equal to }Rk, which is not correct if the parameter
set e is a true subset of }Rk. If e a is an inner point of the original parameter set, then the
vector e == e a + hi is a parameter in e for a given h, for every sufficiently large n,
and the local parameter set converges to the whole of}Rk as n ---+ 00. Then taking the local
parameter set equal to IR k does not cause errors. To give a meaning to the results of this
chapter, the measure Peo+h/ vn may be defined arbitrarily if e a + h I e.
7.2 Expanding the Likelihood
The convergence of the local experiments is defined and established later in this chapter.
First, we discuss the technical tool: a Taylor expansion of the logarithm of the likelihood.
Let Pe be a density of Pe with respect to some measure IL. Assume for simplicity that
the parameter is one-dimensional and that the log likelihood l() (x) == log P() (x) is twice-
differentiable with respect to e, for every x, with derivatives £() (x) and l() (x). Then, for
every fixed x,
P()+h . 1 2" 2
log -ex) == hl()(x) + -h l()(x) + ox(h ).
P() 2
The subscript x in the remainder term is areminder of the fact that this term depends on x
as well as on h. It follows that
n h n 1 h 2 n
O p () + h / -Jii '""" . '""" ..
log (X i )== r;; l()(Xi)+-- l()(Xi)+Remn'
i=l P() v n i=l 2 n i=l
. ...2
Rere the score has mean zero, P()l() == O, and - P()l() == P()l() == l() equals the Fisher infor-
mation for e (see, e.g., section 5.5). Renee the first term can be rewritten as hl::1 n ,(), where
I::1 n ,() == n- 1 / 2 L:7=1 £() (Xi) is asymptotically normal with mean zero and variance l(), by
the central limit theorem. Furthermore, the second term in the expansion is asymptotically
equivalent to _ h2 l(), by the law of large numbers. The remainder term should behave as
0(1 I n) times a sum of n terms and hopefully is asymptotically negligible. Consequently,
under suitable conditions we have, for every h,
O n P()+h/vn 1 2
log (Xi) == hl::1 n ,() - - l()h + O Pe (1).
i=l P() 2
In the next section we see that this is similar in form to the likelihood ratio process of a Gaus-
sian experiment. Because this expansion concems the likelihood process in a neighborhood
of e, we speak of "local asymptotic normality" of the sequence of model s {p()n : e E e}.
The preceding derivation can be made rigorous under moment or continuity conditions
on the second derivative of the log likelihood. Local asymptotic normality was originally
deduced in this manner. Surprisingly, it can also be established under asingle condition that
only involves a first derivative: differentiability of the root density e f--+ ,Jiie in quadratic
mean. This entails the existence of a vector of measurable functions £() == (£(), 1, . . . , £(),k) T
such that
f [ .j Pe+h - ...;pe - hT ili ...;per dlI = o(lIh 11 2 ),
h ---+ O.
(7.1)
94
Local Asymptotic Normality
If this condition is satisfied, then the model (Pe : e E e) is called differentiable in quadratic
mean at e.
U sually, h T le (x) V Pe (x) is the derivative of the map h V Pe+h (x) at h == O for
(almost) every x. In this case
. 1 a VI a
fe(x) == 2 - pe(x) == -logPe(x).
v Pe(x) ae ae
Condition (7.1) does not require differentiability of the map e pe (x) for any single x, but
rather differentiability in (quadratic) mean. Admittedly, the latter is typical1y established by
pointwise differentiability plus a convergence theorem for integrals. Because the condition
is exactly right for its purpose, we establish in the following theorem loeal asymptotic
normality under (7.1). Alemma following the theorem gives easily verifiable conditions in
terms of pointwise derivatives.
7.2 Theorem. Suppose that e is an open subset ofJR.k and that the model (Pe : () E 8)
is differentiable in quadratic mean at (). Then Pefe == O and the Fisher information matrix
le == Pelel exists. Furthermore, for every converging sequence h n ---+ h, as n ---+ 00,
O n Pe+h /-/li 1 T' 1 T
log n (Xi) == r;; h fe(X i ) - -h le h + oPe(I).
i=l pe y n i=l 2
Proof. Given a converging sequence h n ---+ h, we use the abbreviations Pn, p, and g for
Pe+hn/-/li' Pe, and h T le, respectively. By (7.1) the sequenee ( - -}P) converges in
quadratic mean (i.e., in L 2 (/L)) to g -}p. This implies that the sequenee converges in
quadratic mean to -}p. By the continuity of the inner product,
p g = 1 g .jP 2.jP d f-L = lim 1 .jii (.JP;, - .jP)(.JP;, + .JP> d f-L.
The right side equals (1-1) == O for every n, because both probability densities integrate
to 1. Thus Pg == O.
The random variable W ni == 2[ V Pn/ p(X i ) - 1] is with P-probability 1 well defined.
By (7.1)
( nI n )
var W ni - g(Xi) < E(.jiiW ni - g(X i ))2 O,
Et W ni = 2n (I .JP;,.jP df-L - I) = -n 1 [.JP;, - .jPf df-L - Pg2.
(7.3)
Here Pg2 == f g2 dP == h T leh by the detinitions of g and le. If both the means and the
variances of a sequence of random variables converge to zero, then the sequence converges
to zero in probability. Therefore, eombining the preceding pair of displayed equations, we
tind
n 1 n 1
L W ni == - L g(X i ) - _Pg2 + opel).
i=l i=l 4
(7.4)
7.2 Expanding the Likelihood
95
Next, we express the log likelihood ratio in L7=1 W ni through a Taylor expansion of the
logarithm. If we write log(l + x) == x - x2 + x 2 R(2x), then R(x) O as x -+ O, and
n n ( 1 )
log n Pn (Xi) = 2 L log 1 + - W ni
i=1 p i=1 2
n 1 n 1 n
== L W ni - - L Wn + - L Wn R(Wni)' (7.5)
i=1 4 i=1 2 i=1
As a consequence of the right side of (7.3), it is possible to write n Wn == g2 (Xi) + Ani for
random variables Ani such that EIAni I -+ O. The averages An converge in mean and hence
in probability to zero. Combination with the law of large numbers yields
n
"2- -p 2
W ni == (g2)n + An -+ Pg .
i=1
By the triangle inequality followed by Markov's inequality,
nP(IWnil > 8h) < np(g2(Xi) > n8 2 ) +nP(IAnil > n8 2 )
< 8- 2 Pg2{g2 > n8 2 } + 8- 2 EIA ni I O.
The left side is an upperbound for P(maxl i n I W ni I > 8h). Thus the sequence maxl i n
I W ni I converges to zero in probability. By the property of the function R, the sequence
max1:Si n IR(Wni)1 converges in probability to zero as well. The last term on the right
in (7.5) is bounded by max1 i n I R(W ni ) I L7=1 Wn . Thus it is o p (1) O p (1), and converges
in probability to zero. Combine to obtain that
n n 1
n Pn " 2
log -(Xi) == W ni - -Pg + op(l).
i=1 P i=1 4
Together with (7.4) this yields the theorem. .
To establish the differentiability in quadratic mean of specific models requires a conver-
gence theorem for integrals. U sually one proceeds by showing differentiability of the map
() 1---+ Pe (x) for almost every x plus M-equi-integrability (e.g., domination). The following
lemma takes care of most examples.
7.6 Lemma. For every () in an op en subset of"ffi.k let Pe be a M-probability density. Assume
that the map () 1---+ Se (x) == -J pe (x) is continuously differentiablefor every x. Ifthe elements
of the matrix le == J (p e / Pe ) (p / Pe) Pe d Mare well defined a,nd continuous in (), then the
map () ffi is differentiable in quadratic mean (7.1) with le given by p e / Pe.
Proof. By the chain rule, the map () pe (x) == s (x) is differentiable for every x with
gradient Pe == 2sese. Because Se is nonnegative, its gradient Se at a point at which Se == O
must be zero. Conclude that we can write Se == (Pe / Pe) ffi, where the quotient Pe / Pe
may be defined arbitrarily if Pe == O. By assumption, the map () 1---+ le == 4 I ses[ dM is
continuous.
Because the map e 1---+ Se (x) is continuously differentiable, the difference Se+h (x) - Se (x)
can be written as the integral 101 h T Se+uh (x) du of its derivative. By Jensen's (ar Cauchy-
Schwarz's) inequality, the square of this integral is bounded by the integral 101 (h T Se+uh (x))2
96
Local Asymptotic Normality
d u of the square. Conclude that
f ( se+tht - se ) J r ( T. ) 2 1 [ T
t 2 df.L < lo 1 ht SlJ+uth, du df.L = 4 lo 1 ht IlJ+uth, ht du,
where the last equality follows by Fubini's theorem and the definition of le. For ht ---+ h
the right side converges to hT leh == J(h T se)2 dJ-t by the continuity of the map e f--)- le.
By the differentiability of the map e f--)- Se (x) the integrand in
f [ SlJ+th - Se - h T Se r df.L
converges pointwise to zero. The resu1t of the preceding paragraph combined with Propo-
sition 2.29 shows that the integral converges to zero. .
7.7 Example (Exponentialfamilies). The preceding lemma applies to most exponential
family models
pe(x) == d(e)h(x)eQ(e)Tt(x).
An exponential family model is smooth in its natural parame ter (away from the boundary of
the natural parameter space). Thus the maps e -J pe (x) are continuously differentiable
ifthe maps e Q(e) are continuously differentiable and map the parameter set e into the
interior of the natural parameter space. The score function and information matrix equal
fe (x) == Q (t (x) - Ee t (X) ) ,
le == Q cove t(X)(Q )T.
Thus the asymptotic expansion of the local log likelihood is valid for most exponential
families. D
7.8 Example (Location models). The preceding lemma also includes alllocation models
{f (x - e) : e E JR} for apositive, continuously differentiable density f with finite Fisher
information for location
If = f ( ji ) \x) f(x) dx.
The score function le (x) can be taken equal to - (f / / f) (x - e). The Fisher information is
equal to I f for every e and hence certainly continuous in e.
By a refinement of the lemma, differentiability in quadratic mean can also be established
for slightly irregular shapes, such as the Laplace density f (x) == !e- 1xl . For the Laplace
density the map e log f (x - e) fails to be differentiable at the single point e == x.
At other points the derivative exists and equals sign(x - e). It can be shown that the
Laplace location model is differentiable in quadratic mean wi th score fu ncti on fe (x) ==
sign(x - e). This may be pr oved by wr iting the difference -J f(x - h) - -J f(x) as the
integral J !h sign(x - uh) -J f(x - uh) du of its derivative, which is possible even though
the derivative does not exist everywhere. Next the proof of the preceding lemma applies. D
7.9 Counterexample (Uniform distribution). The family of uniform distributions on
[O, eJ is nowhere differentiable in quadratic mean. The reason is that the support of the
7.3 Convergence to a Normal Experiment
97
uniform distribution depends too mueh on the parameter. Differentiability in quadratie
mean (7.1) does not require that all densities pe have the same support. However, restrie-
tion of the integral (7.1) to the set {Pe == O} yields
P e + h (Pe == O) == J Pe+h df.L == o(h 2 ).
Pe=O
Thus, under (7.1) the total mass P e + h (Pe == O) of P e + h that is orthogonal to Pe must
"disappear" as h ---+ O at a rate faster than h 2 .
This is not true for the uniform distribution, beeause, for h > O,
1 1 h
P e + h (Pe == O) == 1 [O,e+h] (x) dx == .
[O,e]C (} + h (} + h
The orthogonal part does eonverge to zero, but only at the rate O (h). D
7.3 Convergence to a Norma} Experiment
The true meaning of loeal asymptotie normality is eonvergenee of the loeal statistieal
experiments to a normal experiment. In Chapter 9 the notion of eonvergenee of statistieal
experiments is introduced in general. In this seetion we bypass this general theory and
establish a direct relationship between the loeal experiments and a normallimit experiment.
The limit experiment is the experiment that eonsists of observing asingle observation X
with the N(h, I e - 1 )-distribution. The log likelihood ratio process of this experiment equals
dN(h, le-I) TIT
log ( 1 ) (X) == h Ie X - -h Ie h .
dN O, Ie- 2
The right side is very similar in form to the right side of the expansion of the log likelihood
ratio log d P;+h/-/11/dP; given in Theorem 7.2. In view of the similarity, the possibility of
a normal approximation is not a eomplete surprise. The approximation in this seetion is
"loeal" in nature: We fix (} and think of
(Pen+h/ -/11 : h E JRk)
as a statistieal model with parameter h, for "known" e. We show that this ean be approxi-
mated by the statistieal model (N(h, le-I) : h E JRk).
Amotivation for studying a loeal approximation is that, usually, asymptotieally, the
"true" parameter ean be known with unlimited preeision. The true statistieal diffieulty is
therefore determined by the nature of the measures Pe for (} in a small neighbourhood of
the true value. In the present situation "small" tums out to be "of size O (1/ -/li)."
A relationship between the models that ean be statistieally interpreted will be deseribed
through the possible (limit) distributions of statisties. For eaeh n, let Tn == Tn (X 1, . . . , X n )
be a statistie in the experiment (P;+h/ -/11 : h E JRk) with values in a fixed Euelidean spaee.
Suppose that the sequenee of statisties T,1 eonverges in distribution under every possible
(Ioeal) parameter:
h
Tn Le h,
,
every h.
98
Local Asymptotic Normality
Here means convergence in distribution under the parameter e + h /,Jn, and Le,h
may be any probability distribution. According to the following theorem, the distributions
{Le,h : h E JRk} are necessarily the distributions of a statistic T in the normal experiment
(N(h, le- l ) : h E JRk). Thus, every weakly converging sequence of statistics is "matched"
by a statistic in the limit experiment. (In the present set-up the vector e is considered known
and the vector h is the statistical parameter. Consequently, by "statistics" Tn and T are
understood measurable maps that do not depend on h but may depend on e.)
This principle of matching estimators is a method to give the convergence of models
a statistical interpretation. Most measures of quality of a statistic can be expressed in the
distribution of the statistic under different parameters. For instance, if a certain hypothesis
is rejected for values of a statistic T,1 exceeding a number c, then the power function
h r--+ P h (Tn > c) is relevant; alternatively, if Tn is an estimator of h, then the mean square
error h r--+ Eh (Tn - h)2, or a similar quantity, determines the quality of Tn. Both quality
measures depend on the laws of the statistics only. The following theorem asserts that as a
function of h the law of a statistic Tn can be well approximated by the law of some statistic
T. Then the quality of the approximating T is the same as the "asymptotic quality" of the
sequence Tn. Investigation of the possible T should reveal the asymptotic performance of
possible sequences Tn. Concrete applications of this princip le to testing and estimation are
given in later chapters.
Aminor technical complication is that it is necessary to allow randomized statistics in
the limit experiment. A randomized statistic T based on the observation X is defined as a
measurable map T == T(X, U) that depends on X but may also depend on an independent
variable U with a uniform distribution on [O, 1]. Thus, the statistician working in the limit
experiment is allowed to base an estimate or test on both the observation and the outcome of
an extra experiment that can be run without knowledge of the parameter. In most situations
such randomization is not useful, but the following theorem would not be true without
it. t
7.10 Theorem. Assume that the experiment (Pe : e E 8) is differentiable in quadratic
mean (7.1) at the point e with nonsingular Fisher information matrix le. Let Tn be statistics
in the experiments (P;+h/ y'n : h E JRk) such that the sequence Tn converges in distribution
under every h. Then there exists a randomized statistic T in the experiment (N (h, le l ) : h E
JRk) such that Tn T for every h.
Proof. For later reference, it is useful to use the abbreviations
Pn,h == P;+h/ y'n'
J == le,
1 "'.
,0.n = ,Jn L.. £e (XJ.
By assumption, the marginals of the sequence (T,1, n) converge in distribution under
h == O; hence they are uniformly tight by Prohorov's theorem. Because marginal tightness
implies joint tightness, Prohorov's theorem can be applied in the other direction to see the
existence of a subsequence of {n} along which
o
(Tn, n) (S, ),
t It is not important that U is uniformly distributed. Any randomization meehanism that is suffieiently rich will
do.
7.3 Convergence to a Normal Experiment
99
jointly, for some random vector (S, ). The vector is necessarily a marginal weak limit
of the sequence n and hence it is N(O, 1)-distributed. Combination with Theorem 7.2
yields
( d Pn,h ) o ( T 1 T )
Tn, log 'v'7 S, h - -h Ih .
d Pn,o 2
In particular, the sequence log dPn,h/dPn,O converges to the normal N( -4hT Ih, h T lh)-
distribution. By Example 6.5, the sequences Pn,h and Pn,o are contiguous. The limit law
L h of Tn under h can therefore be expressed in the joint law on the right, by the general
form of Le Cam's third lemma: For each Borel set B
Lh (B) == E1 B (S)e hT - 4 hT ih.
We need to find a statistic T in the normal experiment having this law under h (for every h),
using only the knowledge that is N (O, 1)-distributed.
By the lemma below there exists a randomized statistic T such that, with U uniformly
distributed and independent of , t
(T( , U), ) r-v (S, ).
Because the random vectors on the left and right sides have the same second marginal
distribution, this is the same as saying that T (8, U) is distributed according to the conditional
distribution of S given == 8, for almost every 8. As shown in the next lemma, this can be
achieved by using the quantile transformation.
Let X be an observation in the limit experiment (N (h, 1-1) : h E JRk). Then IX is under
h == O normally N (O, 1)-distributed and hence it is equal in distribution to . Furthermore,
by Fubini' s theorem,
Ph(T(JX, U) E B) = f P(T(Jx, U) E B) e- (x-h)T J(x-h)
== Eo 1 B (T (I X, U)) e hT ix-4 hT ih.
This equals L h (B), because, by construction, the vec tor (T (1 X, U), IX) has the same
distribution under h == O as (S, ). The randomized statistic T (1 X, U) has law Lh under
h and hence satisfies the requirements. .
detl
dx
(2n ) k
7.11 Lemma. Given arandom vector (S, ) with values in JRd x JRk and an independent
uniformly [O, 1] random variable U (defined on the same probability space), there exists a
jointly measurable map T on JRk x [0,1] such that (T( , U), ) and (S, ) are equal in
distribution.
Proof. For simplicity of notation we only give a construction for d == 2. It is possible
to produce two independent uniform [O, 1] variables Ul and U 2 from one given [O, 1]
variable U. (For instance, construct Ul and U 2 from the even and odd numbered digits in
the decimal expansion of U.) Therefore it suffices to find a statistic T == T ( , U 1, U 2)
such that (T, ) and (S, ) are equal in law. Because the second marginals are equal, it
t The symbol "" means "equal-in-law."
100
Local Asymptotic Normality
suffices to construct T such that T (8, Ul, U 2) is equal in distribution to S given == 8, for
every 8 E R k . Let QI(UII8) and Q2(U218, Sl) be the quantile funetions of the conditional
distributions
pSI I =8
and
p S 21 =8,Sl=SI
,
respectively. These are measurable functions in their two and three arguments, respectively.
Furthermore, QI (U I I8) has law pS] 1 =8 and Q2(U 2 18, Sl) has law pS21 =8,SI=SI, for every
8 and Sl. Set
T (o, Ul, U 2 ) = (Ql (Ulio), Q2 (U2I o , Ql (Ulio))).
Then the first eoordinate QI (U I I8) of T (8, Ul, U 2 ) possesses the distribution pSll =8.
Given that this firstcoordinate equals Sl, the secondcoordinate is distributed as Q2 (U 2 18, Sl),
which has law pS21 =8,SI=SI by construction. Thus T satisfies the requirements. .
7.4 Maximum Likelihood
Maximum likelihood estimators in smooth parametric models were shown to be asymp-
toticalIy normal in Chapter 5. The convergence of the local experiments to a normallimit
experiment gives an insightful explanation of this fact.
By the representation theorem, Theorem 7.10, every sequence of statistics in the local ex-
periments (P;+h/y'ii : h E Rk) is matched in the limit by a statistic in the normal experiment.
Although this does not folIow from this theorem, a sequence of maximum likelihood esti-
mators is typicalIy matched by the maximum likelihood estimator in the limit experiment.
Now the maximum likelihood estilnator for h in the experiment (N(h, IfJ-I) : h E Rk) is the
observation X itself (the mean of a sample of size one), and this is normalIy distributed.
Thus, we should expect that the maximum likelihood estimators h n for the loeal param-
eter h in the experiments (P;+h/ y'ii : h E Rk) converge in distribut on to X. In terms of
the original parameter e, the local maximum likelihood estimator h n is the standardized
maximum likelihood estimator h n == -Jfi(e n - e). Furthermore, the loeal parameter h == O
corresponds to the value e of the original parameter. Thus, we should expect that under
e the sequence -Jfi (en - e) converges in distribution to X under h == O, that is, to the
N (O, Ie- I )-distribution.
As a heuristic explanation of the asymptotic normality of maximum likelihood estimators
the preceding argument is much more insightful than the proof based on linearization of the
seore equation. It also explains why, or in what sense, the maximum likelihood estimator
is asymptoticalIy optimal: in the same sense as the maximum likelihood estimator of a
Gaussian location parameter is optimal.
This heuristie argument cannot be justified under just loeal asymptotie normality, which is
too weak a conneetion between the sequence oflocal experiments and the normallimit exper-
iment for this purpose. Clearly, the argument is valid under the conditions of Theorem 5.39,
because the latter theorem guarantees the asymptotic normality of the maximum likelihood
estimator. This theorem adds a Lipschitz condition on the maps e log PfJ (x), and the
"global" eondition that en is consistent to differentiability in quadratie mean. In the fol-
lowing theorem, we give a direct argument, and also alIow that e is not an inner point of
the parameter set, so that the local parameter spaces may not converge to the fulI space R k .
7.4 Maximum Likelihood
101
Then the maximum likelihood estimator in the limit experiment is a "projection" of X and
the limit distribution of -/fi(e n - 8) may change accordingly.
Let e be an arbitrary subset of IR k and define Hn as the local parameter space Hn
-/fi(8 - e). Then h n is the maximizer over Hn of the random function (or "process")
dp n
h r--+ lo e+hl ft
g dp n
e
If the experiment (Pe : 8 E e) is differentiable in quadratic mean, then this sequence of
processes converges (marginally) in distribution to the process
dN(h, le-I) 1 T 1 T
h r--+ log ( 1 ) (X) == --eX - h) le(X - h) + -X leX.
dN O, le- 2 2
If the sequence of sets Hn converges in a suitable sense to a set H, then we should expect,
under regularity conditions, that the sequence h n converges to the maximizer h of the latter
process over H. This maximizer is the projection of the vector X onto the set H relative
to the metric d(x, y) == (x - y)T le(x - y) (where a "projection" means a closest point); if
H == JRk, this projection reduces to X itself.
An appropriate notion of convergence of sets is the following. Write Hn --* H if H
is the set of all limit s lim h n of converging sequences h n with h n E Hn for every n and,
more over, the limit h == limi h ni of every converging sequence h ni with h ni E H ni for every
i is contained in H.t
7.12 Theorem. Suppose that the experiment (Pe : 8 E e) is differentiable in quadratic
mean at 8 0 with nonsingular Fisher information matrix leo' Furthermore, suppose that for
every el and 8 2 in a neighborhood of 8 0 and a measurable function i with Peof2 < 00,
Ilog Pe\ (x) - log P e 2 (x) I < i(x) 118 1 - 8 2 11.
lf the sequence of maximUln likelihood estimators en is consistent and the sets Hn ==
1 12 "-
-/fi(8 - e o ) converge to a nonempty, convex set H, then the sequence leo -/fi(e n - ( 0 )
converges under e o in distribution to the projection of a standard normal vector onto the
1/2
set leo H.
*Proof. Let G n == -/fi(IPn - Pe o ) be the empirical process. In the proof of Theorem 5.39
it is shown that the map () r--+ log pe is differentiable at 8 0 in L 2 (Pe o ) with derivative
-€e o and that the map 8 r--+ Peo log pe permits a Taylor expansion of order 2 at ()o, with
"second-derivative matrix" - leo' Therefore, the conditions of Lemma 19.31 are satisfied
for me == log Pe, whence, for every M,
Peo+hlft T . 1 T P
sup nIfD n log - h Gn-€e o + - h leoh --* O.
Ilh II::sM peo 2
By Corollary 5.53 the estimators en are -/fi-consistent under ()o.
The preceding display is also valid for every sequence M n that diverges to 00 sufficiently
slowly. Fix such a sequence. By the -/fi-consistency of en, the local maximum likelihood
t See Chapter 16 for examples.
102
Local Asymptotic Normality
estimators h n are bounded in probability and hence belong to the balls of radius M n with
probability tending to 1. Furthermore, the sequence of intersections Hn n ball (O, M n )
converges to H, as the original sets Hn. Thus, we may assume that the h n are the maximum
likelihood estimators relative to local parameter sets Hn that are contained in the balls of
radius M n . Fix an arbitrary closed set F. If h n EF, then the log likelihood is maximal on
F. Renee P(h n E F) is bounded above by
P( TJl) I Peo+h/ vrt TJl) 1 Peo+h/ Vrt )
sup lin og > sup lin og
hEFnH I1 peo hEH I1 Peo
== P ( sup hT!Gn£(}o - hT I(}oh > sup hT!Gn£(}o - hT I(}oh + Opel) )
hEFnH I1 hEH I1
= P( II I(} 1/2!G n £(}o - I :2 (F n Hn) II < II I(} 1/2!Gn£(}o - I :2 Hn II + o p (1)),
by completing the square. By Lemma 7.13 (ii) and (iii) ahead, we can replace Hn by H on
both sides, at the cost of adding a further o p (l)-term and increasing the probability. Next,
by the continuous mapping theorem and the continuity of the map Z 1---+ II z - A II for every
set A, the probability is asymptotically bounded above by, with Z a standard normal vector,
p(IIZ - I \F n H)II < Ilz - I :2HII).
The projection nz of the vector Z on the set I :2 H is unique, because the latter set is
convex by assumption and automatically closed. If the distance of Z to I :2(F n H) is
smaller than its distance to the set I :2 H, then nz must be in I :2(F n H). Consequently,
the probability in the last display is bounded by p(n Z E I :2 F). The theorem follows from
the portmanteau lemma. .
7.13 Lemma. If the sequence of subsets Hn ofIR. k converges to a nonempty set H and
the sequence ofrandom vectors X n converges in distribution to arandom vector X, then
(i) IIX n - Hnll 'V'7IIX - Hil.
(ii) IIX n - Hn n FII > IIX n - H n FII + op(l), for every closed set F.
(iii) IIX n - Hn n GII < IIX n - H n GII + op(l), for every open set G.
Proof. (i). Because the map X Ilx - Hil is (Lipschitz) continuous for any set H,
we have that IIX n - HII'V'7I1X - Hil by the continuous-mapping theorem. Ifwe also show
that II X n - Hn II - II X n - H II O, then the proof is complete after an application of
Slutsky's lemma. By the uniform tightness of the sequence X n , it suffices to show that
Ilx - Hn II Ilx - Hil uniformly for x ranging over compact sets, or equivalently that
Ilxn - Hn II Ilx - H II for every converging sequence X n x.
Forevery fixed vector X n , there exists a vec tor h n E Hn with Ilxn - Hn II > IIxn -h n II-lin.
Unless IIxn - Hn II is unbounded, we can choose the sequence h n bounded. Then every
subsequence of h n has a further subsequence along which it converges, to alimit h in H.
Conclude that, in any case,
liminfllx n - Hnll > liminf!lx n - hnll > Ilx - hil > Ilx - Hil.
Conversely, for every 8 > O there exists h E H and a sequence h n h with h n E Hn and
Ilx - Hil > IIx - hil - 8 == lim Ilxn - hnll - 8 > lim sup Ilxn - Hnll- 8.
7.6 Local Asymptotic Normality
103
Combination of the Iast two displays yields the desired convergence of the sequence
Ilxn - Hnll to Ilx - Hil.
(ii). The assertion is equivalent to the statement p(IIX n - Hn n FII - IIX n - H n FII >
-8) ----+ 1 for every 8 > O. In view of the uniform tightness of the sequence X n , this follows
if lim inf Ilxn - Hn n F II > Ilx - H n F II for every converging sequence X n -+ x. We can
prove this by the method of the first half of the proof of (i), replacing Hn by Hn n F.
(iii). Analogously to the situation under (ii), it suffices to prove that lim sup Ilxn - Hn n
G II < Ilx - H n G II for every converging sequence X n -+ x. This follows as the second
half of the proof of (i). .
*7.5 Limit Distributions under Alternatives
Local asymptotic normality is a convenient tool in the study of the behavior of statistics
under "contiguous alternatives." Under local asymptotic normality,
d pn ( 1 )
lo ()+h/ N __h T I h h T Ih.
g d pn 2 (), ()
()
Therefore, in view of Example 6.5 the sequences of distributions P;+h/ and p are
mutually contiguous. This is of great use in many proofs. With the help of Le Cam's
third lemma it also allows to obtain limit distributions of statistics under the parameters
() + h / ,Jn, once the limit behavior under () is known. Such limit distributions are of interest,
for instance, in studying the asymptotic efficiency of estimators or tests.
The general scheme is as follows. Many sequences of statistics Tn allow an approxima-
tion by an average of the type
1 n
.jii(Tn - fLe) = ,Jn o/e(Xi) + oPo(l).
According to Theorem 7.2, the sequence of log likelihood ratios can be approximated by an
average as well: It is asymptotically equivalent to an affine transformation of n-I 12 L f() (Xi)'
The sequence of j oint averages n-I 12 L ( 0/ () (Xi), f() (Xi)) is asymptoticall y multivariate
normal under () by the central limit theorem (provided o/() has mean zero and finite second
moment). With the heIp of Slutsky's lemma we obtain the joint limit distribution of Tn and
the log likelihood ratios under () :
( dP;+h/ ) () (( O ) ( p()o/()o/J
.j1i(Tn - fLe), log dpn 'v'7 N _lhT 1 h' T T'
() 2 () P() 0/ () h £()
P() o/()h T f() ) ) .
h T I()h
Finally we can apply Le Cam's third Example 6.7 to obtain the limit distribution of
,Jn(T n - IL()) under e + hl,Jn. Concrete examples of this scheme are discussed in later
chapters.
*7.6 Local Asymptotic Normality
The preceding sections of this chapter are restricted to the case of independent, identically
distributed observations. However, the general ideas have a much wider applicability. A
104
Local Asymptotic Norn1alžty
wide variety of models satisfy a general form of loeal asymptotic normality and for that
reason allow a unified treatment. These include models with independent, not identieally
distributed observations, but also models with dependent observations, such as used in time
series analysis or eertain random fields. Beeause loeal asymptotie normality underlies a
large part of asymptotie optimality theory and also explains the asymptotie normality of
eertain estimators, such as maximum likelihood estimators, it is worthwhile to formulate a
general eoneept.
Suppose the observation at "time" n is distributed aeeording to a probability measure
Pn,e, for a parameter 8 ranging over an open subset 8 of IR k .
7.14 Definition. The sequenee of statistieal model s (Pn,e : 8 E 8) is locally asymptoti-
cally normal (LAN) at 8 if there exist matriees r n and le and random veetors n,e such that
n,e N(O, le) and for every eonverging sequenee h n h
d P e -l h 1
log n, +rn Il == h T n,e - - 2 hT leh + 0Pn,e (1).
dPn,e
7.15 Example. If the experiment (Pe : 8 E 8) is differentiable in quadratie mean, then the
sequenee of models (P; : 8 E 8) is loeally asymptotieally normal with norming matriees
r n == l. D
An inspeetion of the proof of Theorem 7.10 readily reveals that this depends on the loeal
asymptotie normality property only. Thus, the loeal experiments
(P n ,e+r,-;l h : h E JRk)
of a locall y asymptoticall y normal sequenee eonverge to the experiment (N (h, I e- 1 ) : h E
JRk), in the sense of this theorem. All results for the ease of i.i.d. observations that are based
on this approximation extend to generalloeally asymptotieally normal models. To illustrate
the wide range of applieations we include, without proof, three examples, two of which
involve dependent observations.
7.16 Example (Autoregressive processes). An autoregressive proeess {X t : t E Z} of or-
der 1 satisfies the relationship Xt == 8X t - 1 + Zt for a sequenee of independent, identieally
distributed variables . . . , Z-1, Zo, ZI, . . . with mean zero and finite varianee. There ex-
ists a stationary solution . . . , X -1 , X o, X 1, . . . to the autoregressive equation if and only if
18 I i- 1. To identify the parameter it is usually assumed that 18 I < 1. If the density of the
noise variables Z j has finite Fisher information for location, then the sequenee of models
eorresponding to observing X] , . . . , X n with parameter set ( -1, 1) is loeally asymptotieally
normal at 8 with norming matriees r n == l.
The observations in this model form a stationary Markovchain. The result extends to
general ergodie Markovchains with smooth transition densities (see [130]). D
7.17 Example (Gaussian time series). This example requires same knowledge of time-
series models. Suppose that at time n the observations are a streteh X 1, . . . , X n from a
stationary, Gaussian time series {X t : t E Z} with mean zero. The eovarianee matrix of n
7.6 Local Asymptotic Normality
105
consecutive variables is given by the (Toeplitz) matrix
TnUe) = (i: ei(t-s)). Je(A) dA) _ .
s,t-l,...,n
The function fe is the spectral density of the series. It is convenient to let the parameter
enter the model through the spectral density, rather than directly through the density of the
observations.
Let Pn,e be the distribution (on JRn) of the vector (XI, . . . , X n ), a normal distribution
with mean zero and covariance matrix Tn (fe). The periodogram of the observations is the
function
2
1 n .
In (A) == - L Xte ltA .
2nn t=l
Suppose that fe is bounded away from zero and infinity, and that there exists a vector-valued
function ie : JR 1--+ JRd such that, as h ---+ O,
f T . 2 ( 2 )
[fe+h - fe - h fe fe] dA == o Ilhll .
Then the sequence of experiments (Pn,e : e E e) is locally asymptotically normal at e with
r n == ,jfi,
f ie
n e == - (In - EeIn)- dA,
, 4n fe
1 f . 'T
le == - fefe dA.
4n
The proof is elementary, but involved, because it has to deal with the quadratic forms in
the n-variate normal density, which involve vectors whose dimension converges to infinity
(see [30]). D
7.18 Example (Almost regular densities). Consider estimating a location parameter e
based on a sample of size n from the density f(x - e). If f is smooth, then this model is
differentiable in quadratic mean and hence locally asymptotically normal by Example 7.8.
If f possesses points of discontinuity, or other strong irregularities, then a locally asymptot-
ically normal approximation is impossible. t Examples of densities that are on the boundary
between these "extremes" are the triangular density f (x) == (1 - Ix I) + and the gamma
density f(x) == xe- x 1 { x > O}. These yield model s that are locally asymptotically normal,
but with norming rate y' n log n rather than . The existence of singularities in the density
makes the estimation of the parameter e easier, and hence a faster rescaling rate is necessary.
(For the triangular density, the true singularities are the points -1 and 1, the singularity at
O is statistically unimportant, as in the case of the Laplace density.)
For a more general result, consider densities f that are absolutely continuous except pos-
sibly in small neighborhoods Ul, . . . , U k of finitely many fixed points Cl, . . . , Ck. Suppose
that f! / ,JJ is square- integrable on the complement of U jU j, that f (c j) == O for every j,
and that, for fixed constants al, . . . , ak and bI, . . . , bk, each of the functions
X 1--+ f(x) - (a j 1{x < Cj} + b j 1{x > cj})lx - cjl,
x E U j ,
t See Chapter 9 for some examples.
106
Local Asymptotic Normality
is twice continuously differentiable. If L (a j + b j) > O, then the model is locally asymp-
totically normal at () == O with, for V n equal to the interval (n- 1 / 2 (logn)-1/4, (log n)-l)
around zero,t
lo == L(aj + bj),
]
fln,o= 1 tt ( l{X;-CjEV n } - [ f(X+Cj)dX ) .
,J n log n i=l j=l Xi - Cj iVI! X
r n == J n log n,
The sequence n,O may be thought of as "asymptotically sufficient" for the Iocal parameter h.
Its detinition of n,O shows that, asymptotically, all the "information" about the parameter is
contained in the observations falling into the neighborhoods V n + C j. Thus, asymptotically,
the problem is determined by the po ints of irregularity.
The remarkable rescaling rate ,J n log n can be explained by computing the Hellinger
distance between the densities f (x - ()) and f (x) (see section 14.5). D
Notes
Local asymptotic normality was introduced by Le Cam [92], apparently motivated by the
study and construction of asymptotically similar tests. In this paper Le Cam detines two
sequences of models (Pn,e : () E 8) and (Qn,e : () E 8) to be differentially equivalent if
sup IIP n ,e+h/J11 - Qn,e+h/J11 II --+ O,
hEK
for every bounded set K and every (). He next shows that a sequence of statistics Tn in a given
asymptotically differentiable sequence of experiments (roughly LAN) that is asymptotically
equivalent to the centering sequence n,e is asymptotically sufticient, in the sense that the
original experiments and the experiments consisting of observing the Tn are differentially
equivalent. After some interpretation this gives roughly the same message as Theorem 7.10.
The latter is a concrete example of an abstract result in [95], with a different (direct) proof.
PROBLEMS
1. Show that the Poisson distribution with mean e satisfies the conditions of Lemma 7.6. Find the
information.
2. Find the Fisher information for location for the normal, logistic, and Laplace distributions.
3. Find the Fisher information for location for the Cauchy distributions.
4. Let f be a density that is symmetric about zero. Show that the Fisher infofmation matrix (if it
exists) of the location scale family f ((x - IL) / (5 ) / (5 is diagonal.
5. Find an explicit expression for the oP e (l)-term in Theorem 7.2 in the case that pe is the density
of the N (e, 1 )-distribution.
6. Show that the Laplace location family is differentiable in quadratic mean.
t See, for example, [80, pp. 133-139] for a pro of, and also a diseussion of other almost regular situations. For
instance, singularities of the form f (x) "-' f (ej) + Ix - ej 1 1 / 2 at points ej with f (e i) > O.
Problems
107
7. Find the form of the score function for a location-scale family f ((x - f-L) / a ) / a with parameter
() == (f-L, a) and apply Lemma 7.6 to find a sufficient condition for differentiabi1ity in quadratic
mean.
8. Investigate for which parameters k the location family f (x - ()) for f the gamma(k, 1) density
is differentiable in quadratic mean.
9. Let Pn,() be the distribution of the vector (XI, . . . , X n ) if {X t : t E Z} is a stationary Gaussian
time series satisfying Xt == ()X t -l + Zt for a given number I()I < 1 and independent standard
normal variables Zt. Show that the model is locally asymptotically norma!.
10. Investigate whether the log normal family of distributions with density
1 e - 2 2 (log(x- )-J-t)2 1 {x > }
a (x - )
is differentiable in quadratic mean with respect to () == ( , f-L, a).
8
Efficiency of Estimators
One purpose of asymptotic statistics is to compare the performance of
estimators for large sample sizes. This chapter discusses asymptotic lower
bounds for estimation in locally asymptotically normal models. These
show, among others, in what sense maximum likelihood estimators are
asymptotically efficient.
8.1 Asymptotic Concentration
Suppose the problem is to estimate 1/1' (8) based on observations from amodel governed by
the parameter 8. What is the best asymptotic performance of an estimator sequence Tn for
1/1'(8)?
To simplify the situation, we shalI in most of this chapter assume that the sequence
-Jli(T,'l -1/1'(8)) converges in distribution under every possible value of 8. Next we rephrase
the question as: What are the best possible limit distributions? In analogy with the Cramer-
Rao theorem a "best" limit distribution is referred to as an asymptotic lower bound. Under
certain restrictions the normal distribution with mean zero and covariance the inverse Fisher
information is an asymptotic lower bound for estimating 1/1' (8) == 8 in a smooth parametric
model. This is the main result of this chapter, but it needs to be qualified.
The notion of a "best" limit distribution is understood in terms of concentration. If the
limit distribution is a pri ori assumed to be normal, then this is usualIy translated into asymp-
totic unbiasedness and minimum variance. The statement that -Jli (Tn - 1/1' (8) ) converges in
distribution to a N(IL(8), (J2(8) )-distribution can be roughly understood in the sense that
eventualIy Tn is approximately normally distributed with mean and variance given by
1L(8) (J2(8)
1/1'(8) + - and
-Jli n
Because Tn is meant to estimate 1/1' (8), optimal choices for the asymptotic mean and variance
are 1L(8) == O and variance (J2(8) as small as possible. These choices ensure not only that
the asymptotic mean square error is small but also that the limit distribution N (IL (8), (J 2 (8) )
is maximally concentrated near zero. For instance, the probability of the interval (-a, a)
is maxirnized by choosing IL (8) == O and (J 2 (8) minimal.
We do not wish to assume a pri ori that the estimators are asymptotically norma!. That
normallimits are best will actually be an interesting conclusion. The concentration of a
general limit distribution Le cannot be measured by mean and variance alone. Instead, we
108
8.1 Asymptotic Concentratian
109
can employ a variety of concentration meas ures, such as
f x 2 dLe(x);
f Ixl dLe(x);
f l{lxl > a} dLe(x); f (Ixl J\ a) dLe(x).
Alimit distribution is "good" if quantities of this type are small. More generally, we
focus on minimizing J f dLe for a given nonnegative function f. Such a function is called
a loss function and its integral J f d Le is the asymptotic risk of the estimator. The method
of measuring concentration (or rather lack of concentration) by means of loss functions
applies to one- and higher-dimensional parameters alike.
The following example shows that a detinition of what constitutes asymptotic optimality
is not as straightforward as it might seemo
8.1 Example (Hodges' estimator). Suppose that Tn is a sequence of estimators for a real
parameter 8 with standard asymptotic behavior in that, for each 8 and certain limit distri-
butions Le,
viii(T n - 8)! Le.
As a specitic example, let Tn be the mean of a sample of size n from the N (8, l)-distribution.
Define a second estimator Sn through
Sn = { n
if 11;11 > n- 1 / 4
if ITnl < n- 1 / 4 '
If the estimator Tn is already close to zero, then it is changed to exactly zero; otherwise it
is left unchanged. The truncation point n- 1 / 4 has been chosen in such away that the limit
behavior of Sn is the same as that of Tn for every 8 =j:. O, but for 8 == O there appears to be a
great improvement. Indeed, for every r n ,
o
r n Sn "v'7 O
r-/rf /1'- e T
y n .)n - tj) "v'7 Le,
e :t O.
To see this, note first that the probability that Tn falls in the interval (8 - M n -1/2, 8 + M n -1/2)
converges to Le( -M, M) for most M and hence is arbitrarily close to 1 for M and n
sufficiently large. For 8 =j:. O, the interval s (8 - Mn- 1 / 2 , 8 + Mn- 1 / 2 ) and (_n- 1 / 4 , n- 1 / 4 )
are centered at different places and eventually disjoint. This implies that truncation will
rarely occur: Pe (Tn == Sn) ---+ 1 if 8 =j:. O, whence the second assertion. On the other hand the
interval (-Mn- 1 / 2 , Mn- 1 / 2 ) is contained in the interval (_n- 1 / 4 , n- 1 / 4 ) eventually. Renee
under 8 == O we have truncation with probability tending to 1 and hence Po (Sn == O) ---+ 1;
this is stronger than the first assertion.
At first sight, Sn is an improvement on Tn. For every 8 =j:. O the estimators behave
the same, while for e == O the sequence Sn has an "arbitrarily fast" rate of convergence.
Rowever, this reasoning is a bad use of asymptotics.
Consider the concrete situation that Tn is the mean of a sample of si ze n from the
normal N (8, 1)-distribution. It is well known that Tn == X is optimal in many ways for
every fixed n and hence it ought to be asymptotically optimal also. Figure 8.1 shows
why Sn == X I { I X I > n-I /4} is no improvement. It shows the graph of the risk function
8 Ee (Sn - 8)2 for three different values of n. These functions are close to 1 on most
110
Efficiency of Estimators
LO
T"""
o
T"""
o
LO
-2
-1
o
2
Figure 8.1. Quadratic risk functions of the Hodges estimator based on the means of samples of
size 10 (dashed), 100 (dotted), and 1000 (solid) observations from the N(e, l)-distribution.
of the domain but possess peaks close to zero. As n ---+ 00, the locations and widths of
the peaks converge to zero but their heights to infinity. The conclusion is that Sn "buys"
its better asymptotic behavior at () == O at the expense of erratic behavior close to zero.
Because the values of e at which Sn is bad differ from n to n, the erratic behavior is not
visible in the pointwise limit distributions under fixed e. D
8.2 Relative Efficiency
In order to choose between two estimator sequences, we compare the concentration of their
limit distributions. In the case of normallimit distributions and convergence rate .Jn, the
quotient of the asymptotic variances is a good numerical measure of their relative efficiency.
This number has an attractive interpretation in terms of the numbers of observations needed
to attain the same goal with each of two sequences of estimators.
Let v ---+ 00 be a "time" index, and suppase that it is required that, as 1) ---+ 00, our
estimator sequence attains mean zero and variance 1 (or l/v). Assume that an estimator Tn
based on n observations has the property that, as n ---+ 00,
(Tn - ljJ (e) ) N (O, a 2 (e) ) .
Then the requirement is to use at time 1) an appropriate number nv of observations such
that, as v ---+ 00,
(Tnv - ljJ(e)) N(O, 1).
Given two available estimator sequences, let n v , 1 and n v ,2 be the nurnbers of observations
8.3 Lower Boundfor Experiments
111
needed to meet the requirement with each of the estimators. Then, if it exists, the limit
1 . nv,2
Im-
V--HX) nv, I
is called the relative efficiency of the estimators. (In general, it depends on the para-
meter 8.)
Because -JV(Tnv - 1/1(e)) can be written as -Jv/nv (Tnv - 1/1(8)), it follows that
necessarily nv -+ 00, and also that nv/v -+ a 2 (8). Thus, the relative efficiency of two
estimator sequences with asymptotic variances a;(e) is just
1 . nv,2/v _ ai(8)
Im - 2 .
v---+oo nv,I/v al (8)
If the value of this quotient is bigger than 1, then the second estimator sequence needs
proportionally that many observations more than the first to achieve the same (asymptotic)
precIsIon.
8.3 Lower Bound for Experiments
It is certainly impossible to give a nontriviallower bound on the limit distribution of a
standardized estimator ,Jn (Tn - 1/1 (e)) for asingle 8. Hodges' example shows that it is
not even enough to consider the behavior under every e, pointwise for all e. Different
values of the parameters must be taken into account simultaneously when taking the limit
as n -+ 00. We shalI do this by studying the performance of estimators under parameters
in a "shrinking" neighborhood of a fixed 8.
We consider parameters 8 + h/,Jn for 8 fixed and h ranging over k and suppose that,
for certain limit distributions Le,h,
.Jn( Tn - 1/1 (e + :n )) e+'!1../ri Le,h' every h.
(8.2)
Then Tn can be considered a good estimator for 1/1 (8) if the limit distributions Le,h are
maximally concentrated near zero. If they are maximally concentrated for every h and
some fixed 8, then Tn can be considered locally optimal at e. Unless specified otherwise,
we assume in the remain der of this chapter that the parameter set e is an open subset of
k, and that 1/1 maps e into m. The derivative of 8 1--+ 1/1 (8) is denoted by 1/1e.
Suppose that the observations are a sample of size n from a distribution Pe. If Pe depends
smoothly on the parameter, then
(P;+h/../ri: h E k) "" (N(h, leI): h E k),
as experiments, in the sense of Theorem 7.10. This theorem shows which limit distributions
are possible and can be specialized to the estimation problem in the following way.
8.3 Theorem. Assume that the experiment (Pe : 8 E 8) is difJerentiable in quadratic
mean (7.1) at the point e with nonsingular Fisher information matrix le. Let 1/1 be dif-
ferentiable at 8. Let Tn be estimators in the experiments (P 'l+h/ -/ii : h E :tRk) such that
112
Efficiency of Estimatars
(8.2) holds for every h. Then there exists a randomized statistic T in the experiment
(N(h, l()-I): h E JRk) such that T -1y()h has distribution L(),h!or every h.
Proof. Apply Theorem 7.10 to Sn == -J11(Tn - 1jJ(e)). In view of the detinition of L(),h
and the differentiability of 1jJ, the sequence
Sn = 01(Tn - 0/(8 + )) + 01(0/(8 + ) - 0/(8»)
converges in distribution under h to L(),h * 8;Pe h ' where *8 h denotes a translation by h.
According to Theorem 7.10, there exists a randomized statistic T in the normal experiment
such that T has distribution L(),h * 8;Pe h for every h. This satisties the requirements. .
This theorem shows that for most estimator sequences Tn there is a randomized estimator
T such that the distribution of -J11 (Tn - 1jJ (e h / -J11)) under e + h / -J11 is, for large n,
approximately equal to the distribution of T -1jJ()h under h. Consequently the standardized
distribution of the best possible estimator Tn for 1jJ (e + h / -J11) is approximately equal to the
standardized distribution of the best possible estimator T for 1jJ() h in the limit experiment. If
we know the best estimator T for 1jJ()h, then we know the "locally best" estimator sequence
Tn for 1fr ( e) .
In this way, the asymptotic optimality problem is reduced to optimality in the experiment
based on one observation X from a N(h, 1()-l)-distribution, in which e is known and h
ranges over JRk. This experiment is simple and easy to analyze. The observation itself is
. .
the customary estimator for its expectation h, and the natural estimator for 1jJ() h is 1jJ() X.
This has several optimality properties: It is minimum variance unbiased, minimax, best
equivariant, and Bayes with respect to the noninformative prior. Some of these properties
are reviewed in the next section.
. .
Let us agree, at least for the moment, that 1jJ() X is a "best" estimator for 1jJ() h. The
distribution of 1y()X -1y()h is normal with zero mean and covariance 1y()I()-I1y()T for every h.
The parameter h == O in the limit experiment corresponds to the parameter e in the original
problem. We conclude that the "best" limit distribution of -J11(Tn - 1jJ(e)) under e is the
N (O, 1y() 1()-I1y() T)-distribution.
This is the main result of the chapter. The remaining sections discuss several ways of
making this reasoning more rigorous. Because the expression 1p() 1()-I1tre T is precisely the
Cramer-Rao lower bound for the covariance of unbiased estimators for 1jJ (e), we can think
of the resuits of this chapter as asymptotic Cramer-Rao bounds. This is helpful, even though
it does not do justice to the depth of the present results. For instance, the Cramer-Rao bound
in no way suggests that normallimiting distributions are best. Also, it is not completely
true that an N (h, l()-I )-distribution is "best" (see section 8.8). We shall see exactly to what
extent the optimality statement is false.
8.4 Estimating Normal Means
According to the preceding section, the asymptotic optimality problem reduces to optimality
in a normallocation (or "Gaussian shift") experiment. This section has nothing to do with
asymptotics but reviews some facts about Gaussian models.
8.4 Estimating Normal Means
113
Based on asingle observation X from a N(h, h)-distribution, it is required to estimate
Ah for a given matrix A. The covariance matrix h is assumed known and nonsingular. It
is well known that AX is minimum variance unbiased. It will be shown that AX is also
best-equivariant and mini max for many loss functions.
A randomized estimator T is called equivariant-in-Iaw for estimating Ah if the distri-
bution of T - Ah under h does not depend on h. An example is the estimator AX, whose
"invariant law" (the law ofAX - Ah under h) is the N (O, Ah A T)-distribution. The follow-
ing proposition gives an interesting characterization of the law of general equivariant-in-law
estimators: These are distributed as the sum ofAX and an independent variable.
8.4 Proposition. The nulI distribution L of any randomized equivariant-in-Iaw estimator
of Ah can be decomposed as L == N(O, AhA T) * M for some probability measure M. The
only randomized equivariant-in-Iaw estimator for which M is degenerate at O is AX.
The measure M can be interpreted as the distribution of a noise factor that is added to
the estimator AX. If no noise is best, then it follows that AX is best equivariant-in-law.
Amore precise argument can be made in terms ofloss functions. In general, convoluting
a measure with another measure decreases its concentration. This is immediately clear in
terms of variance: The variance of a sum of two independent variables is the sum of the
variances, whence convolution increases variance. For normal measures this extends to
all "bowl-shaped" symmetric loss functions. The name should convey the form of their
graph. Formally, a function is defined to be bowl-shaped if the sublevel sets {x : l (x) < c}
are convex and symmetric about the origin; it is called subconvex if, moreover, these sets
are closed. A loss function is any function with values in [O, (0). The following lemma
quantifies the loss in concentration under convolution (for a proof, see, e.g., [80] or [114].)
8.5 Lemma (Anderson's lemma). For any bowl-shaped loss function l on ffi.k, every prob-
ability measure M on ffi.k, and every covariance matrix h
f £dN(O, I;) < f £d[N(O, I;) * MJ.
N ext consider the minimax criterion. According to this criterion the "best" estimator,
relative to a given loss function, minimizes the maximum risk
sup Ehl(T - Ah),
h
over all (randomized) estimators T. For every bowl-shaped loss function l, this leads again
to the estimator AX.
8.6 Proposition. For any bowl-shaped loss function l, the maximum risk of any random-
ized estimator T of Ah is bounded below by Eol(AX). Consequently, AX is a minimax
estimator for Ah. If Ah is real and E o (AX)2l(AX) < 00, then AX is the only minimax
estimator for Ah up to changes on sets of probability zero.
Proofs. For a proof of the uniqueness of the minimax estimator, see [18] or [80]. We
prove the other assertions for subconvex loss functions, using a Bayesian argument.
114
Efficiency of Estimatars
Let H be arandom vec tor with a normal N (O, A)-distribution, and consider the original
N(h, b)-distribution as the conditional distribution of X given H == h. The randomization
variable U in T (X, U) is constructed independently of the pair (X, H). In this notation, the
distribution of the variable T - AH is equal to the "average" of the distributions of T - Ah
under the different values of h in the original set-up, averaged over husing a N (O, A)- "prior
distribution."
By a standard calculation, we find that the "a posteriori" distribution, the distribution
of H given X, is the normal distribution with mean (b- 1 + A -1 )-1 b-I X and covariance
matrix (b -1 + A -1 ) -1. Define the random vectors
W A == T - A(b- 1 + A -1)-lb- 1 X, GA == -A(H - (b- 1 + A -1)-lb- 1 x)
These vectors are independent, because W A is a function of (X, U) only, and the condi-
tional distribution of GA given X is normal with mean O and covariance matrix A(b- 1 +
A -1) -1 AT, independent of X. As A == AI for a scalar A 00, the sequence G A converges
in distribution to a N(O, AbA T)-distributed vector G. The sum of the two vectors yields
T - AH, for every A.
Because a supremum is larger than an average, we obtain, where on the left we take the
expectation with respect to the original model,
sup Ehl(T - Ah) > El(T - AH) == El(G A + W A ) > El(G A ),
h
by Anderson' s lemma. This is true for every A. The lim inf of the right side as A ---+ 00 is
at least El(G), by the portmanteau lemma. This concludes the proof that AX is minimax.
If T is equivariant-in-law with invariant law L, then the distribution of GA + W A
T - AH is L, for every A. It follows that
f eit T X dL(x) = Ee itT GA Ee itTWA , every t.
As A 00, the left side remains fixed; the first factor on the right side converges to the
characteristic function of G, which is positive. Conclude that the characteristic functions of
W A converge to a continuous function, whence W A converges in distribution to some vector
W, by Levy's continuity theorem. By the independence of GA and W A for every A, the
sequence (GA, W A) converges in distribution to a pair (G, W) of independent vectors with
marginal distributions as before. N ext, by the continuous- mapping theorem, the distribution
of GA + W A , which is fixed at L, "converges" to the distribution of G + W. This proves
that L can be written as a convolution, as claimed in Proposition 8.4.
If T is an equivariant-in-law estimator and t(X) == E(T(X, U)I x), then
Eh(t - AX) == Eh(T - AX) == Eh(T - Ah) - Eh(AX - Ah)
is independent of h. By the completeness of the normallocation family, we conclude that
t - AX is constant, almost surely. If T has the same law as AX, then the constant is zero.
Furthermore, T must be equal to its projection t almost surely, because otherwise it would
have a bigger second moment than t == AX. Thus T == AX almost surely. .
8.6 Almost-Everywhere Convolution Theorem
115
8.5 Convolution Theorem
An estimator sequence Tn is called regular at e for estimating a parameter 0/ (e) if, for
every h,
Jn( Tn - \I1(e + )) e+ fi Le.
The probability measure Le may be arbitrary but should be the same for every h.
A regular estimator sequence attains its limit distribution in a "locally uniform" manner.
This type of regularity is common and is often considered desirable: A small change in
the parameter should not change the distribution of the estimator too much; a disappearing
small change should not change the (limit) distribution at all. However, some estimator
sequences of interest, such as shrinkage estimators, are not regular.
In term s of the limit di s tributi on s Le,h in (8.2), regularity is exact1y that all Le,h are
equal, for the given e. According to Theorem 8.3, every estimator sequence is matched
by an estimator T in the limit experiment (N(h, le-I): h E JR.k). For a regular estimator
sequence this matching estimator has the property
. h
T - o/eh r-v Le, every h.
(8.7)
Thus a regular estimator sequence is matched by an equivariant-in-law estimator for o/eh.
Amore informative name for "regular" is asymptotically equivariant-in-law.
It is now easy to determine a best estimator sequence from among the regular estima-
tor sequences (a best regular sequence): It is the sequence Tn that corresponds to the best
. .
equivariant-in-Iaw estimator T for o/eh in the limit experiment, which is o/eX by Proposi-
tion 8.4. The best possible limit distribution of a regular estimator sequence is the law of
this estimator, a N (O, e Ie- I e T )-distribution.
The characterization as a convolution of the invariant laws of equivariant-in-law estima-
tor s carries over to the asymptotic situation.
8.8 Theorem (Convolution). Assume that the experiment (Pe : e E 8) is differentiable
in quadratic mean (7.1) at the point e with nonsingular Fisher information matrix le.
Let 0/ be differentiable at e. Let Tn be an at e regular estimator sequence in the experi-
ments (Pen: e E 8) with limit distribution Le. Then there exists a probability measure Me
such that
( . l' T )
Le == N O, o/e le o/e * Me.
-In particular, if Le has covariance matrix be, then the matrix :Ee - e le- I e T is nonnegative-
definite.
Proof. Apply Theorem 8.3 to conclude that Le is the distribution of an equivariant-in-Iaw
estimator T in the limit experiment, satisfying (8.7). Next apply Proposition 8.4. .
8.6 Almost-Everywhere Convolution Theorem
Hodges' example shows that there is no hope for a nontrivial lower bound for the limit
distribution of a standardized estimator sequence -Jfi (Tn - 0/ (e)) for every e. It is always
116
Efficiency of Estimatars
possible to improve on a given estimator sequence for selected parameters. In this section
it is shown that improvement over an N(O, VreIe-1VreT)-distribution can be made on at most
a Lebesgue nun set of parameters. Thus the possibilities for improvement are very much
restricted.
8.9 Theorem. Assume that the experiment (Pe : e E 8) is differentiable in quadratic
mean (7.1) at every e with nonsingular Fisher information matrix le. Let 1/1' be differentiable
at every e. Let Tn be an estimator sequence in the experiments (P; : () E 8) such that
,jfi(Tn - 1/1'ce)) converges to alimit distribution Le under every (). Then there exist
probability distributions Me such that for Lebesgue almost every ()
( . l' T )
Le == N O, 1/1'e Ie- 1/1e * Me.
. l' T
In particular, if Le has covariance matrix be, then the matrix he - 1/1'e I e- 1/1'e lS
nonnegative definite for Lebesgue almost every ().
The theorem follows from the convolution theorem in the preceding section combined
with the following remarkable lemma. Any estimator sequence with limit distributions is
automatically regular at almost every e along a subsequence of {n}.
8.10 Lemma. Let Tn be estimators in experiments CPn,e : e E 8) indexed by a measurable
subset e of k. Assume that the map () 1---+ Pn,e CA) is measurable for every measurable
set A and every n, and that the map e 1---+ 1/1 ce) is measurable. Suppose that there exist
distributions Le such that for Lebesgue almost every e
r n (Tn - 1/1' «()) ) ! Le.
Then for every Yn ---+ O there exists a subsequence of {n} such that, for Lebesgue almost
every «(), h), along the subsequence,
( ) e+ h
r n Tn - 1/1'«() + Yn h ) Le.
Proof. Assume without loss of generality that e == k; otherwise, fix some ()a and let
Pn,e == Pn,e o for every () not in 8. Write Tn,() == r n (Tn - 1/1«())). There exists a countable
collection :F of uniformly bounded, left- or right-continuous functions f such that weak
convergence of a sequence of maps Tn is equivalent to Ef (Tn) ---+ f f dL for every f E :F. t
Suppose that for every f there exists a subsequence of {n} along which
Ee+Ynh!(Tn,e+Yn h ) --+ J ! dL e ,
A 2k - a.e. (e, h).
Even in case the subsequence depends on f, we can, by a diagonalization scheme, con-
struct a subsequence for which this is valid for every f in the countable set :F. Along this
subsequence we have the desired convergence.
t For eontinuous distributions L we can use the indieator funetions of cells (-00, eJ with c ranging over Qk. For
general L replaee every such indieator by an approximating sequenee of continuous funetions. Altematively,
see, e.g., Theorem 1.12.2 in [146]. Also see Lemma 2.25.
8.7 Local Asymptotic Minimax Theorem
117
Settinggn(e) == E()f(Tn,()) andg(e) == J f dL(), weseethatthelemmaisprovedoncewe
have established the folIowing assertion: Every sequence ofbounded, measurable functions
gn that converges almost everywhere to alimit g, has a subsequence along which
gn (e + Ynh) -+ g(e),
A 2k - a. e. ( e, h).
We may assume without loss of generality that the function g is integrable; otherwise we
first multiply each gn and g with a suitable, fixed, positive, continuous function. It should
also be verified that, under our conditions, the functions gn are measurable.
Write p for the standardnormal density onJR k and Pn for the density of the N(O, I +Y; 1)-
distribution. By Scheffć's lemma, the sequence Pn converges to p in LI. Let 8 and H den ote
independent standard normal vectors. Then, by the triangle inequality and the dominated-
convergence theorem,
Elgn(8 + Yn H ) - g(8 + YnH)1 = !Ign(u) - g(u) IPn(U)du -+ O.
Secondly for any fixed continuous and bounded function g 8 the sequence El g 8 (8 + Yn H) -
g 8 (e) I converges to zero as n -+ 00 by the dominated convergence theorem. Thus, by the
triangle inequality, we obtain
Elg(8 + Yn H ) - g(8)1 < ! 19 - gel(u) (Pn + p)(u)du + 0(1)
= 2 ! 19 - gel(u) p(u) du + 0(1).
Because any measurable integrable function g can be approximated arbitrarily closely in
LI by continuous functions, the first term on the far right side can be made arbitrarily smalI
by choice of g 8' Thus the left side converges to zero.
By combining this with the preceding display, we see that Elgn (8 + Yn H ) - g(8) I -+ O.
In other words, the sequence offunctions (e, h) 1---+ gn(e + Ynh) - g(e) converges to zero
in mean and hence in probability, under the standard normal measure. There exists a
subsequence along which it converges to zero almost surely. .
*8.7 Local Asymptotic Minimax Theorem
The convolution theorems discussed in the preceding sections are not completely satisfying.
The convolution theorem designates a best estimator sequence among the regular estimator
sequences, and thus imposes an a priori restriction on the set of permitted estimator se-
quences. The almost-everywhere convolution theorem imposes no (serious) restriction but
yields no information about some parameters, albeit a nulI set of parameters.
This section gives a third attempt to "prove" that the normal N (O, 1y() I()-I 1y() T)-distribution
is the best possible limit. It is based on the minimax criterion and gives a lower bound for the
maximum risk over a small neighborhood of a parameter e. In fact, it bounds the expression
lim lim inf sup E(),R- ( vn (Tn - VJ (el) ) ) .
o O n oo 1I()'-()II<o
This is the asymptotic maximum risk over an arbitrarily smalI neighborhood of e. The
following theorem concerns an even more refined (and smalIer) version of the local maxi-
mum risk.
118
Efficiency of Estimators
8.11 Theorem. Let the experiment (P() : e E 8) be differentiable in quadratic me an (7.1) at
e with nonsingular Fisher information matrix I(). Let 1/1 be differentiable at e. Let Tn be
any estimator sequence in the experiments (P; : e E }Rk). Then for any bowl-shaped loss
function -€
S Pl f EHh/Ji1l(vn(Tn -1/r(e + ))) > f ldN(O, ele-l eT).
Bere the first supremum is taken over all finite subsets I of IR k .
Proof. We only give the proof under the further assumptions that the sequence ,Jfi (Tn -
1/1(e») is uniformly tight under e and that -€ is (lower) semicontinuous. t Then Prohorov's
theorem shows that every subsequence of {n} has a further subsequence along which the
vectors
( vn(Tn -1/r(e)), .Jn L> e(Xi))
converge in distribution to alimit under e. By Theorem 7.2 and Le Cam's third lemma, the
sequence ,Jfi ( Tn -1/1 (e) ) converges in law also under every e + h / ,Jfi along the subsequence.
By differentiability of 1/1 , the same is true for the sequence ,Jfi (Tn -1/1 (e + h / ,Jfi) ), whe ce
(8.2) is satisfied. By Theorem 8.3, the distributions L(),h are the distributions of T - 1/1()h
under h for a randomized estimator T based on an N(h, 1()-I)-distributed observation. By
Proposition 8.6,
sup Eh-€(T -lfr()h) > Eo-€(lfr()X) == f l dN(O, lfr()I()-llfr()T).
h EIR k
It suffices to show that the left side of this display is a lower bound for the left side of the
theorem.
The complicated construction that defines the asymptotic minimax risk (the lim inf sand-
wiched between two suprema) requires that we apply the preceding argument to a carefully
chosen subsequence. Place the rational vectors in an arbitrary order, and let lk consist of
the first k vectors in this sequence. Then the left side of the theorem is larger than
R:== limliminfSUPE()+h/ -€ ( vn ( Tn-1/1 ( e+ ))) .
k-+oo n-+oo hEh V n
There exists a subsequence {nk} of {n} such that this expression is equal to
lim sup E()+h/ k -€ ( vnk ( Tnk - 1/1 ( e + ) ) ) .
k-+oo hEh ,Jnk
We apply the preceding argument to this subsequence and find a further subsequence along
which Tn satisfies (8.2). For simplicity of notation write this as {n'} rather than with a
double subscript. Because -€ is nonnegative and lower semicontinuous, the portmanteau
lemma gives, for every h,
l 1!} f EHh/Rl( (Tn' -1/r(e + ))) > f ldLe,h'
t See, for example, [146, Chapter 3.11] for the general result, which can be proved along the same lines, but using
a eompactification deviee to induee tightness.
8.8 Shrinkage Estimators
119
Every rational vector h is contained in lk for every sufticiently large k. Conclude that
R > sup f l dLe,h == sup Ehl(T -1fre h ).
hEQk hEQk
The risk function in the supremum on the right is lower semicontinuous in h, by the
continuity of the Gaussian location family and the lower semicontinuity of l. Thus
the expression on the right does not change if (Ql is replaced by JRk. This concludes the
proof. .
*8.8 Shrinkage Estimators
The theorems of the preceding sections seem to prove in a variety of ways that the best
possible limit distribution is the N (O, e le 1 e T)-distribution. At closer inspection, the
situation is more complicated, and to a certain extent optimality remain s amatter of taste,
asymptotic optimality being no exception. The "optimal" normallimit is the distribution
of the estimator l/1e X in the normallimit experiment. Because this estimator has several
optimality properties, many statisticians consider it best. N evertheless, one might prefer a
Bayes estimator or a shrinkage estimator. With a changed perception of what constitutes
"best" in the limit experiment, the meaning of "asymptotically best" changes also. This
becomes particularl y clear in the example of shrinkage estimators.
8.12 Example (Shrinkage estimator). Let XI, . . . , X n be a sample from a multivariate
normal distribution with mean e and covariance the identity matrix. The dimension k of
the observations is assumed to be at least 3. This is essential! Consider the estimator
X n
Tn == X n - (k - 2) - .
nllX n l12
Because X n converges in probability to the mean (), the second term in the detinition of Tn
is O p (n-I) if e =I- O. In that case -Jn(Tn - X n ) converges in probability to zero, whence
the estimator sequence Tn is regular at every () =I- O. For () == hi -Jn, the variable -JnXn is
distributed as a variable X with an N (h, l)-distribution, and for every n the standardized
estimator -Jn (Tn - hi -Jn) is distributed as T - h for
X
T(X)==X-(k-2) 2 '
IIXII
This is the Stein shrinkage estimator. Because the distribution of T - h depends on h, the
sequence Tn is not regular at e == O. The Stein estimator has the remarkable property that,
for every h (see, e.g., [99, p. 300]),
Eh II T - h 11 2 < Eh IIX - h 11 2 == k.
It fol1ows that, in term s of joint quadratic loss R(x) == Ilx 11 2 , the locallimit distributions
LO,h of the sequence -Jn(Tn - hi -Jn) under () == hi -Jn are all better than the N(O, l)-limit
distribution of the best regular estimator sequence X n' D
The example of shrinkage estimators shows that, depending on the optimality criterion, a
normal N(O, ele-l e T)-limit distribution need not be optimal. In this light, is itreasonable
120
Efficiency of Estimatars
to uphold that maximum likelihood estimators are asymptotically optimal? Perhaps not. On
the other hand, the possibility of improvement over the N (O, 1;re le- 1 1;re T )-limit is restricted
in two important ways.
First, improvement can be made only on a null set of parameters by Theorem 8.9.
Second, improvement is possible only for specialloss functions, and improvement for one
loss function necessarily implies worse performance for other loss functions. This follows
from the next lemma.
Suppose that we require the estimator sequence to be locally asymptotically minimax for
a given loss function -€ in the sense that
S Pli S P fE8+h/ftf( 0l( Tn -1/f(e + ))) < J fdN(O, te 1 e - 1 te T ).
This is a reasonable requirement, and few statisticians would challenge it. The following
lemma shows that for one-dimensional parameters 1/1 (e) local asymptotic minimaxity for
even asingle loss function implies regularity. Thus, if it is required that all coordinates of a
certain estimator sequence be locally asymptotically minimax for some loss function, then
the best regular estimator sequence is optimal without competition.
8.13 Lemma. Assume that the experiment (Pe : e E 8) is dijferentiable in quadratic mean
(7.1) at e with nonsingular Fisher information matrix le. Let 1/1 be a real-valued map
that is dijferentiable at e. Then an estimator sequence in the experiments (P(j : e E k)
can be locally asymptotically minimax at e for a bowl-shaped loss function -€ such that
O < f x 2 l(x) dN(O, 1;r el e- 1 1;re T )(x) < 00 only ifTn is best regular at e.
Proof. We only give the proof under the further assumption that the sequence -Vii(Tn -
1/1 (e)) is uniforml y tight under e. Then by the same arguments as in the proof of Theo-
rem 8.11, every subsequence of {n} has a further subsequence along which the sequence
-vii (Tn - 1/1 (e + h / -vii)) converges in distribution under e + h / -vii to the distribution
Le,h of T -1;reh under h, for a randomized estimator T based on an N(h, le- 1 )-distributed
observation. Because Tn is locally asymptotically minimax, it follows that
sup Eh-€(T -1;re h ) == sup J -€ dLe,h < J -€ dN(O, te l e - 1 te T ).
hE k hE k
Thus T is a minimax estimator for 1/1eh in the limit experiment. By Proposition 8.6,
T == 1/1eX, whence Le,h is independent of h. .
*8.9 Achieving the Bound
If the convolution theorem is taken as the basis for asymptotic optimality, then an estimator
sequence is best if it is asymptotically regular with a N (O, 1;re le- 1 1;re T )-limit distribution.
An estimator sequence has this property if and only if the estimator is asymptotically linear
in the score function.
8.14 Lemma. Assume that the experiment (Pe : e E 8) is differentiable in quadratic
me an (7.1) at e with nonsingular Fisher information matrix le. Let 1/1 be differentiable at
8.9 Achieving the Bound
121
(). Let Tn be an estimator sequence in the experiments (P; : () E JRk) such that
C ( ) 1 . 1 .
y n Tn -1jJ(()) == 1/1 e1 e- fe(X i ) + oP e (1).
n . 1
1=
Then Tn is best regular estimator for 1/1(()) at (). Conversely, every best regular estimator
sequence satisfies this expansion.
Proof. The sequence b..n,e == n- 1 / 2 L fe (Xi) converges in distribution to a vec tor b..e with
a N (O, le )-distribution. By Theorem 7.2 the sequence log d P;+h/-.;nl d P; is asymptotically
equivalent to h T b..n,e - hT leh. If Tn is asymptotically line ar, then -Jfi(Tn - 1/1(())) is
asymptotically equivalent to the function lf;e le- 1 b..n,e. Apply Slutsky' s lemma to find that
( C ( ) dPe+h/vn ) e ( . 1 TIT )
V n Tn - 1/J(e) , log d P; -V-+ 1/J(} I(}- b.(}, h b.(} -"2 h I(}h
(( ) ( . l' T
O 1/1e le- 1/1e
rovN 1 T .
-"ih le h 1/1e hT
1jJe h ))
h T le h .
The limit distribution of the sequence -Jfi(Tn -1/1(())) under () + hi -Jfi follows by Le Cam's
third lemma, Example 6.7, and is normal with mean lf;eh and covariance matrix lf;e1e-1lf;eT.
Combining this with the differentiability of 1/1, we obtain that Tn is regular.
Next suppose that Sn and Tn are both best regular estimator sequences. By the same
argument s as in the proof of Theorem 8.11 it can be shown that, at least along subsequences,
the joint estimators (Sn, Tn) for (1/1 (()), 1/1 (())) satisfy for every h
( ( Sn -1/J(e + :n )). (Tn -1/J(e + :n ))) (}+ (S - (}h, T - (}h),
for a randomized estimator (S, T) in the normal-limit experiment. Because Sn and Tn are
best regular, the estimators S and T are best equivariant-in-law. Thus S == T == 1/1eX almost
surely by Proposition 8.6, whence -Jfi(Sn - Tn) converges in distribution to S - T == O.
Thus every two best regular estimator sequences are asymptotically equivalent. The
second assertion of the lemma follows on applying this to Tn and the estimators
1 . -1
Sn = 1/J (e) + -Jfi 1/J(} I(} b. n ,(}.
Because the parameter () is known in the local experiments (P;+h/vn : h E JRk), this indeed
defines an estimator sequence within the present context. It is best regular by the first part
of the lemma. .
Under regularity conditions, for instance those of Theorem 5.39, the maximum likeli-
hood estimator en in a parametric model satisfies
A 1 -1'
(()n - ()) == -Jfi le fe (Xi) + o Pe (1).
n . 1
1=
Then the maximum likelihood estimator is asymptotically optimal for estimating () in terms
of the convolution theorem. By the delta method, the estimator 1/1 (en) for 1/1 (()) can be seen
122
Efficiency of Estimators
to be asymptotically linear as in the preceding theorem, so that it is asymptotically regular
and optimal as well.
Actually, regular and asymptotically optimal estimators for f) exist in every parametric
model (Pe : () E 8) that is differentiable in quadratic mean with nonsingular Fisher infor-
mation throughout 8, provided the parameter f) is identifiable. This can be shown using
the discretized one-step method discussed in section 5.7 (see [93]).
*8.10 Large Deviations
Consistency of an estimator sequence Tn entails that the probability of the event
d ( Tn , 1/f ( ())) > £ tends to zero under (), for every £ > O. This is a very weak require-
ment. One method to strengthen it is to make £ dependent on n and to require that the
probabilities P e (d (Tn, 1/f (())) > £ n) converge to O, or are bounded away from 1, for a given
sequence £n O. The resu1ts of the preceding sections address this question and give very
precise lower bounds for these probabilities using an "optimal" rate £n == r;; 1 , typically
-1/2
n .
Another method of strengthening the consistency is to study the speed at which the
probabilities Pe (d(Tn, 1/f(())) > £) converge to O for a fixed £ > O. This method appears
to be of less importanee but is of some interest. Typically, the speed of convergence is
exponential, and there is a precise lower bound for the exponential rate in term s of the
Kullback - Leibler information.
We consider the situation that Tn is based on arandom sample of size n from a distribution
Pe, indexed by a parameter () ranging over an arbitrary set 8. We wish to estimate the value
of a function 1/f : 8 f-+ II]) that takes its values in a metric space.
8.15 Theorem. Suppose that the estimator sequence Tn is consistent for 1/f (f)) under every
(). Then, for every £ > O and every ()o,
l imSUP- lOgPeo ( d(Tn' 1/f(()o)) > £ ) < inf -Pelog pe o .
n-+oo ne: deo/ce), 0/(e o ))>8 pe
Proof. If the right side is infinite, then there is nothing to proveo The Kullback - Leibler
information - Pe log Peo / pe can be finite only if Pe « Peo. Renee, it suffices to prove that
- Pe log Peo / pe is an upper bound for the left side for every f) such that Pe « Pe o and
d(1/f(()),1/f(()o)) > £. The variable An == (n-I) 2:7=1 10g(Pe!Peo) (Xi) is well defined
(possibly -(0). For every constant M,
Pe o (d (Tn' 1/f(()o)) > lO) > Pe o (d(Tn, 1/f(()o)) > lO, An < M)
> Ee 1 {d (Tn, 1/f(()o)) > lO, An < M }e- nAn
> e nM Pe(d(Tn, 1/f(()o)) > lO, An < M).
Take logarithms and mu1tiply by - (1/ n) to conclude that
- log Pe o (d(Tn, 1/f(()o)) > lO) < M - log Pe (d(T n , 1/f(()o)) > lO, An < M).
For M > Pe log Pe / Peo' we have that Pe (An < M) 1 by the law of large numbers.
Furthermore, by the consistency of 1',1 for 1/f (()), the probability Pe (d ( Tn, 1/f (f)o)) > £)
Problems
123
converges to 1 for every e such that d ( 1/1 (e), 1/1 (e o )) > £. Conclude that the probability in
the right side of the preceding display converges to 1, whence the lim sup of the left side is
bounded by M. .
Notes
Chapter 32 of the famous book by Cramer [27] gives a rigorous proof of what we now
know as the Cramer-Rao inequality and next goe s on to define the asymptotic efficiency of
an estimator as the quotient of the inverse Fisher information and the asymptotic variance.
Cramer defines an estimator as asymptotically efficient if its efficiency (the quotient men-
tioned previously) equals one. These definitions lead to the conclusion that the method of
maximum likelihood produces asymptotically efficient estimators, as already conjectured
by Fisher [48, 50] in the 1920s. That there is a conceptual hole in the definitions was clearly
realized in 1951 w hen Hodges produced his example of a superefficient estimator. Not long
after this, in 1953, Le Cam proved that superefficiency can occur only on a Lebesgue nulI
set. Our present result, almost without regularity conditions, is based on later work by Le
Cam (see [95].) The asymptotic convolution and minimax theorems were obtained in the
present form by Hajek in [69] and [70] after initial work by many authors. Our present
proofs folIow the approach based on limit experiments, initiated by Le Cam in [95].
PROBLEMS
1. Calculate the asymptotic relative efficiency of the sample mean and the sample median for
estimating e, based on a sample of size n from the normal N (f), 1) distribution.
2. As the previous problem, but now for the Laplace distribution (density p(x) == e-Ixl).
3. Consider estimating the distribution function P(X < x) at a fixed point x based on a sample
XI, ..., X n from the distribution of X. The "nonparametric" estimator is n- 1 #(Xi < x). If it
is known that the true underlying distribution is normal N (e, 1), another possible estimator is
<I> (x - X). Calculate the relative efficiency of these estimators.
4. Calculate the relative efficiency of the enlpirical p-quantile and the estimator ep -1 (p) Sn + X n
for the estimating the p-th quantile of the distribution of a sample from the normal N CI.i-, (j2)_
distribution.
5. Consider estimating the population variance by either the sample variance S2 (which is unbiased)
or else n-I I: 7= 1 (Xi - X )2 == (n - l)/n S2. Calculate the asymptotic relative efficiency.
6. Calculate the asymptotic relative efficiency of the sample standard deviation and the interquartile
range (corrected for unbiasedness) for estimating the standard deviation based on a sample of
si ze n from the normal N (/1, (j2)-distribution.
7. Given a sample of size n from the uniform distribution on [O, eJ, the maximum X (n) of the
observations is biased downwards. Because Ee (e - X (n)) == Ee X (1), the bias can be removed by
adding the minimum of the observations. Is X (1) + X (n) a good estimator for e from an asymptotic
point of view?
8. Consider the Hodges estimator Sn based on the mean of a sample from the N (e, l)-distribution.
(i) Show that "jii(Sn - en) - 00, if en --+ O in such away that n 1/4e n --+ O and n 1/2e n --+ 00.
(ii) Show that Sn is not regular at e == o.
124
Efficiency of Estimatars
(iii) Show that sUP-o<e<o Pe(J1lISn - el > kn) 1 for every kn that converges to infinity
sufficiently slowly.
9. Show that a loss function l: JR JR is bowl-shaped if and only ifit has the form l(x) == lo(lxl)
for a nondecreasing function lo.
10. Show that a function of the form l(x) == lo(llx II) for a nondecreasing function lo is bowl-shaped.
11. Prove Anderson's lemma for the one-dimensional case, for instance by calculating the derivative
of f l(x + h) dN(O, l)(x). Does the proof generali ze to higher dimensions?
12. What does Lemma 8.13 imply about the coordinates of the Stein estimator. Are they good
estimators of the coordinates of the expectaction vector?
13. All results in this chapter extend in a straightforward manner to generallocally asymptotically
normal models. Formulate Theorem 8.9 and Lemma 8.14 for such models.
9
Limits of Expe riments
A sequence of experiments is defined to converge to alimit experiment if
the sequence of likelihood ratio processes converges marginally in dis-
tribution to the likelihood ratio process of the limit experiment. Alimit
experiment serves as an approximation for the converging sequence of
experiments. This generalizes the convergence of locally asymptotically
normal sequences of experiments considered in Chapter 7. Several ex-
amples of nonnormallimit experiments are discussed.
9.1 Introduction
This chapter introduces a notion of convergence of statistical models or "experiments" to
alimit experiment. In this notion a sequence of models, rather than just a sequence of
estimators or tests, converges to a limit. The limit experiment serves two purposes. First,
it provides an absolute standard for what can be achieved asymptotically by a sequence of
tests or estimators, in the form of a "lower bound": N o sequence of statistical procedures
can be asymptotically better than the "best" procedure in the limit experiment. For instance,
the best limiting power function is the best power function in the limit experiment; a best
sequence of estimators converges to a best estimator in the limit experiment. Statements
of this type are true irrespective of the precise meaning of "best." A second purpose of
alimit experiment is to explain the asymptotic behaviour of sequences of statistical pro-
cedures. For instance, the asymptotic normality or (in)efficiency of maximum likelihood
estimators.
Many sequences of experiments converge to normaIlirnit experiments. In particular,
the local experiments in a given locally asymptotically normal sequence of experiments,
as considered in Chapter 7, converge to a normal location experiment. The asymptotic
representation theorem given in the present chapter is therefore a generalization of Theo-
rem 7.10 (for the LAN case) to the general situation. The importance of the general concept
is illustrated by several examples of non-Gaussian limit experiments.
In the present context it is customary to speak of "experiment" rather than model, al-
though these terms are interchangeable. Formally an experiment is a measurable space
(X, A), the sample space, equipped with a collection of probability measures (P h : h E H).
The set of probability measures serves as a statistical model for the observation, written as
X. In this chapter the parameter is denoted by h (and not (j), because the results are typi-
cally applied to "loe al" parameters (such as h == ,.Jii((j - (ja)). The experiment is denoted
125
126
Limits of Experiments
by (X, A, P h : h E H) and, if there can be no rnisunderstanding about the sample space,
also by (Ph : h EH).
Given a fixed parameter ha EH, the likelihood ratio process with base ha is formed as
( dPh ex) ) = ( (X) ) .
dPh o hEB PhO hEB
Each likelihood ratio process is a (typically infinite-dimensional) vector of random variables
dP h / dPho (X). According to the results of section 6.1, the right side of the display is P ho -
almost surely the same for any given densities Ph and Ph o with respect to any measure Jl.
Because we are interested only in the laws under Ph o of finite subvectors of the likelihood
processes, the nonuniqueness is best left unresolved.
9.1 Definition. A sequence En == (X n , An, Pn,h : h E H) of experiments converges to a
limit experiment E == (X, A, P h : h E H) if, for every finite subset I C H and every ha EH,
( dP h ) h ( dP h )
n, (X n ) -eX) .
dPn,h o hE! dPh o hE!
The objects in this display are random vectors of length I I I. The requirement is that each
of these vectors converges in law, under the assumption that ha is the true parameter, in the
ordinary sense of convergence in distribution in JR!. This type of convergence is sametimes
called marginal weak convergence: The finite-dimensional marginal distributions of the
likelihood processes converge in distribution to the corresponding marginals in the limit
experiment.
Because a weak limit of a sequence of random vectors is unique, the marginal distributions
of the likelihood ratio process of alimit experiment are unique. The limit experiment its elf
is not unique; even its sample space is not uniquely determined. This causes no problems.
Two experiments of which the likelihood ratio processes are equal in marginal distributions
are called equivalent or of the same type. Many examples of equivalent experiments arise
through sufficiency.
9.2 Example (Equivalence by sufficiency). Let S : X f-+ Y be a statistic in the statistical
experiment (X, A, P h : h E H) with values in the measurable space (Y, B). The experiment
of image laws (Y, B, P h o S-I : h E H) corresponds to observing S. If S is a sufficient
statistic, then this experiment is equivalent to the original experiment (X, A, P h : h E H).
This may be proved using the N eyman factorization criterion of sufficiency. This shows
that there exist measurable functions gh and f such that Ph (x) == gh (S (x) ) f (x), so that
the likelihood ratio Ph/ Ph o (X) is the function gh/ gho (S) of S. The likelihood ratios of the
measures P h o S-I take the same form.
Consequently, if (Ph : h E H) is alimit experiment, then so is (Ph o S-I: h EH). A very
simple example that we encounter frequently is as follows: For a given invertible matrix I
the experiments (N(Ih, 1): h E JRd) and (N(h, 1-1): h E JRd) are equivalent. O
9.2 Asymptotic Representation Theorem
In this section it is shown that alimit experiment is always statistically easier than a
given sequence. Suppose that a sequence of statistical problems involves experiments
9.3 Asymptotic Normality
127
En == (Pn,h : h E H) and statistics Tn. For instance, the statistics are test statistics for testing
certain hypotheses conceming the parameter h, or estimatars of same function of h. Most
of the quality measures of the procedures based on the statistics Tn can be expressed in their
laws under the different parameters. For simplicity we assume that the sequence of statistics
Tn converges under a given parameter h in distribution to alimit L h, for every parameter
h. Then the asymptotic quality of the sequence Tn may be judged from the set of limit
laws {Lh : h EH}. According to the following theorem the only possible sets of limit laws
are the laws of randomized statistics in the limit experiment: Every weakly converging se-
quence of statistics converges to a statistic in the limit experiment. One consequence is that
asymptotically no sequence of statistical procedures can be better than the best procedure in
the limit experiment. This is true for every meaning of "good" that is expressible in terms of
laws. In this way the limit experiment obtains the character of an asymptotic lower bound.
We assume that the limit experiment E == (P h : h E H) is dominated: This requires the
existence of a (J - finite measure ji.; such that P h « ji.; for every h. Recall that a randomized
statistic T in the experiment (X, A, P h : h E H) with values in TRk is a measurable map
T : Xx [O, 1] TRk for the product (J-field A x Borel sets on the space Xx [O, 1]. Its law
under h is to be computed under the product measure P h x uniform[O, 1].
9.3 Theorem. Let En == (X n , An, Pn,h : h E H) be a sequence of experiments that conver-
ges to a dominated experiment E == (X, A, P h : h EH). Let Tn be a sequence of statistics
in En that converges in distribution for every h. Then there exists a randomized statistic T
in E such that Tn T for every h.
Proof. The proof of the theorem starting from the definition of convergence of experi-
ments is long and can best be broken up into parts of independent interest. This goes beyond
the scope of this book.
The proof for the case of local asymptotic normal sequences of experiments is given in
Chapter 7. (It is shown in Theorem 9.4 that such a sequence of experiments converges to
a Gaussian location experiment.) Many other examples can be treated by the same method
of proof. t .
9.3 Asymptotic Normality
As in much of statisties, normallimits are of prime importanee. In Chapter 7 a sequence of
statistical models (P n ,() : e E 8) indexed by an open subset 8 c }Rd is defined to be locally
asymptotically normal at e if the log likelihood ratios log dP n ,()+r;;l h n / dP n ,() allow a certain
quadratic expansion. This is shown to be valid in the case that P n ,() is the distribution of a
sample of size n from a smooth parametrie model. Sueh experiments converge to simple
normallimit experiments if they are reparametrized in terms of the "loeal parameter" h.
This follows from the following theorem.
9.4 Theorem. Let En == (Pn,h : h E H) be a sequence of experiments indexed by a subset H
of TR d (with O EH) such that
dPn h TIT
log , == h n - -h ih + op 0 (1),
dPn,o 2 n,
t For a proof of the general theorem see, for instance, [141].
128
Limits of Experiments
for a sequence of statistics n that converges weakly under h == O to a N (O, J) -distribution.
Then the sequence £n converges to the experiment (N(Jh, I) : h EH).
Proof. The log likelihood ratio process with base ha for the normal experiment has
coordinates
dN (I h, I) T 1 T 1 T
log (X) == (h - ha) X - -h Ih + -ha Iha.
dN(Iha, I) 2 2
If I is nonsingular, then this follows by simple algebra, because the left side is the quotient
of two normal densities. The case that I is singular perhaps requires some thought.
By the assumption combined with Slutsky's lemma, the sequence log Pn}l/ Pn,a is under
h == O asymptotically normal with mean _ hT Ih and variance h T Ih). This implies con-
tiguity of the sequences of measures Pn,h and Pn,a for every h, by Example 6.5. Therefore,
the probability of the set on which one of Pn,a, Pn,h, or Pn,h o is zero converges to zero.
Outside this set we can write
1 Pn,h 1 Pn,h 1 Pn,ho
og- == og- - og-o
Pn,h o Pn,a Pn,a
Because this is true with probability tending to 1, the difference between the left and the right
sides converges to zero in probability. Apply the (local) asymptotic normality assumption
twice to obtain that
Pn h T 1 T 1 T
log ==(h-ha) n--h Jh+-haJha+o Ph (1).
Pn,h o 2 2 11, o
On comparing this to the expression for the normallikelihood ratio process, we see that
it suffices to show that the sequence n converges under ha in law to X: In that case
the vec tor (Pn,h/Pn,ho)hEI converges in distribution to (dN(Ih, J)jdN(O, I)(X))hEI' by
Slutsky's lemma and the continuous-mapping theorem.
By assumption, the sequence ( n, h6 n) converges in distribution under h == O to
a vector ( , h6 ), where is N (O, I)-distributed. By local asymptotic normality and
Slutsky's lemma, the sequence of vectors ( n, log Pn,h o / Pn,a) converges to the vector
( , h6 - h6 Iha). In other words
( Pn,h o ) a (( O ) ( I
n, log - 'v'-t N _lhT Ih ' h T I
Pn,a 2 a a a
I ha ) )
h6 J ha .
By the Gaussian form of Le Cam's third lemma, Example 6.5, the sequence n converges
in distribution under ha to a N (I ha, I)-distribution. This is equal to the distribution of X
under ha. .
9.5 Co rollary. Let 8 be an open subset ofJR.. d , and let the sequence of statistical models
(PnJ) : e E 8) be locally asymptotically normal at e with norlning lnatrices r n and a
nonsingular matrix le. Then the sequence of experiments (P n ,e+r;;lh : h E JR.d) converges to
the experiment (N (h, Ii I ) : h E JR..d).
9.4 Uniform Distribution
129
9.4 Uniform Distribution
The model consisting of the uniform distributions on [O, e] is not differentiable in quadratic
mean (see Example 7.9.) In this case an asymptotically normal approximation is impossible.
Instead, we have convergence to an exponential experiment.
9.6 Theorem. Let P; be the distribution of arandom sample of size n from a uniform
distribution on [O, eJ. Then the sequence of experiments (P;-h/n : h E JR) converges for
each fixed e > O to the experiment consisting of observing one observation from the shifted
exponential density z e-(z-h)/ tJ l{z > h}/e. t
ProoJ. If Z is distributed according to the given exponential density, then
dPZ e-(Z-h)je I { Z > h }/ e
(Z) == == e(h-ho)/tJ 1 {Z > h}
dPz e-(Z- h o)/el{Z > ha}/e '
ho
almost surely under ha, because the indicator 1 {z > ha} in the denominator equals 1 almost
surely if ha is the true parameter.
The joint density of arandom sample XI, . . . , X n from the uniform [O, e] distribution
can be written in the form (lje)n 1 {X(n) < eJ. The likelihood ratios take the form
dP;-h/n (e - h/n)-nl{X(n) < e - hin}
(XI, . . . , X n ) ==
dP;-ho/n (e - h a /n)-nl{X(n) < e - ha/n}
U nder the parameter e - ha j n, the maximum of the observations is certainly bounded above
bye - ha/n and the indicator in the denominator equals 1. Thus, with probability 1 under
e - ha/n, the likelihood ratio in the preceding display can be written
(e(h-hO)/tJ + 0(1)) 1{-n(X(n) - e) > h}.
By direct calculation, -n (X (n) -e) Z. By the continuous-mapping theorem and Slutsky's
lemma, the sequence of likelihood processes converges under e - ha/n marginally in
distribution to the likelihood process of the exponential experiment. .
Along the same lines it may be proved that in the case of uniform distributions with
both endpoints unknown alimit experiment based on observation of two independent ex-
ponential variables pertains. These types of experiments are completely determined by the
discontinuities of the underlying densities at their left and right endpoints. It can be shown
more generally that exponentiallimit experiments are obtained for any densities that have
jumps at one or both of their endpoints and are smooth in between. For densities with
discontinuities in the middle, or weaker singularities, other limit experiments pertain.
The convergence to alimit experiment combined with the asymptotic representation
theorem, Theorem 9.3, allows one to obtain asymptotic lower bounds for sequences of
estimators, much as in the locally asymptotically normal case in Chapter 8. We give only
one concrete statement.
t Define Pe arbitrarily for e < o.
130
Limits of Experiments
9.7 Corollary. Let Tn be estimators based on a sample XI, . . . , X n from the uniform
distribution on [O, eJ such that the sequence n (Tn - e) converges under e in distribution
to a limit Le, for every e. Then for Lebesgue almost-every e we have J Ix I d Le (x) >
EIZ - med ZI and J x 2 dL e (x) > E(Z - EZ)2 for the random variable Z exponentially
distributed with mean e.
Proof (Sketch). By Lemma 8.10, the estimator sequence Tn is automatically almost reg-
ular in the sense that n (T n - e + h I n) converges under e - h I n in distribution to Le for
Lebesgue almost every e and h, at least along a subsequence. Thus, it is matched in the
limit experiment by an equivariant-in-law estimator for almost every e. More precisely, for
almost every e there exists a randomized statistic Te such that the law of Te (Z + h, U) - h
does not depend on h (if Z is exponentially distributed with mean e). By classical statistical
decision the ory the given lower bounds are the (constant) risks of the best equivariant-in-law
estimators in the exponentiallimit experiment in terms of absolute error and mean-square
error loss functions, respectively. .
In view of this lemma, the maximum likelihood estimator X (n) is asymptotically ineffi-
cient. This is not surprising given its bi as downwards, but it is encouraging for the present
approach that the small bias, which is of the order 1 I n, is visible in the "first -order" asymp-
totics. The bi as can be corrected by a multiplicative factor, which, unfortunately, must
depend on the loss function. The sequences of estimators
n + log 2
X (n) and
n
n+1
X(n)
n
are asymptotically efficient in term s of absolute value and quadratic loss, respectively.
9.5 Pareto Distribution
The Pareto distributions are a two-parameter family of distributions on the real line with
parameters a > O and IL > O and density
alLa
x 1---+ - l l{x > IL}.
x a +
This density is smooth in a, but it resembles a uniform distribution as discussed in the
preceding section in its dependence on IL. The limit experiment consists of a combination
of a normal experiment and an exponential experiment.
The likelihood ratios for a sample of size n from the Pareto distributions with parameters
(a + g/ , IL + hin) and (a + ga/ , IL + haln), respectively, is equal to
( a+g/ ) n (lL+hln)na+vng ( n ) (gO-g)/vn { h }
Xi 1 X(l) > IL + -
a+go/ (p, + ho/n)na+.fiigo 1] n
( 1 g2 + g2 )
= exp (g - gO) n 2: a 2 o + 0(1) (e(h-h o )a/f1. + 0(1») 1 {Zn > h}.
9.6 Asymptotic Mixed Normality
131
Here, under the parameters (ex + go I -Jli, JL + ho I n), the sequence
1 n ( X. 1 )
tl n == - - L log - -
-Jli i=l JL a
converges weakly to a normal distribution with mean gola 2 and variance 1/a 2 ; and the
sequence Zn == n(X(l) - JL) converges in distribution to the (shifted) exponential distri-
bution with mean JLla + ho and variance (JLla)2. The two sequences are asymptotically
independent. Thus the likelihood is a product of a locally asymptotically normal and a
"locally asymptotically exponential" factor. The locallimit experiment consists of observ-
ing a pair (tl, Z) of independent variables tl and Z with a N (g, ( 2 ) -distribution and an
exp(al JL) + h-distribution, respectively.
The maximum likelihood estimators for the parameters a and JL are given by
n
an , and /-Ln == X(1).
= L7=1 1o g(X;j X(1))
The sequence -Jli (an - a) converges in distribution under the parameters (a + g I -Jli, JL +
h I n) to the variable tl - g. Because the distribution of Z does not depend on g, and
follows a normallocation model, the variable tl can be considered an optimal estimator
for g based on the observation (tl, Z). This optimality is carried over into the asymptotic
optimality of the maximum likelihood estimator an. A precise formulation could be given
in terms of a convolution or a minimax theorem.
On the other hand, the maximum likelihood estimator for JL is asymptotically inefficient.
Because the sequence n (P-n - JL - h I n) converges in distribution to Z - h, the estimators
/-Ln are asymptotically biased upwards.
9.6 Asymptotic Mixed Normality
The likelihood ratios of some models allow an approximation by a two-term Taylor ex-
pansion without the linear term being asymptotically normal and the quadratic term being
deterministic. Then a generalization of local asymptotic normality is possible. In the most
important example of this situation, the linear term is asymptotically distributed as a mixture
of normal distributions.
A sequence of experiments (P n ,() : e E 8) indexed by an open subset 8 of JRd is called
locally asymptotically mixed normal at e if there exist matrices Yn,() ---+ O such that
dP 1
lo n,()+Yn.ehn == h T tl - _h T J h + o ( 1 )
g dP n,() 2 n,() Pn.e'
n,()
for every converging sequence h n ---+ h, and random vectors tl n ,() and random matrices J n ,()
such that (tl n ,(), J n ,()) (tl(), J()) for arandom vec tor such that the conditional distribution
of () given that J() == J is normal N (O, J).
Locally asymptotically mixed normal is often abbreviated to LAMN. Locally asymp-
totically normal, or LAN, is the special case in which the matrix J() is deterministic. Se-
quences of experiments whose likelihood ratios allow a quadratic approximation as in the
preceding display (but without the specific limit distribution of (tl n ,(), J n ,())) and that are
132
Limits of Experiments
such that Pn.e+Yn,eh <J [> Pn.e are called locally asymptotically quadratic, or LAQ. We note
that LAQ or LAMN requires much more than the mere existence of two derivatives of the
likelihood: There is no reason why, in general, the remainder would be negligible.
9.8 Theorem. Assume that the sequence of experiments (Pn,e : 8 E 8) is locally asymptot-
ically mixed normal at 8. Then the sequence of experiments (Pn,e+Yn,e h : h E IR d ) converges
to the experiment consisting of observing a pair ( , 1) such that 1 is marginally distributed
as le for every h and the conditional distribution of given 1 is normal N (1 h, 1).
Prool Write Pe,h for the distribution of ( , 1) under h. Because the marginal distribution
of 1 does not depend on h and the conditional distribution of given 1 is Gaussian
dPe,h ( , 1) == dN (1 h, 1) ( ) == e(h-ho)T - hT Jh+ hr; Jh o .
dPe,h o dN(lh o ,l)
By Slutsky's lemma and the assumptions, the sequence dPn,e+Yn,eh/dPn,e converges under
e in distribution to exp(h T e - ihT leh). Because the latter variable has mean one, it
follows that the sequences of distributions Pn,e+Yn,e h and Pn,e are mutually contiguous. In
particular, the probability under 8 that dPn,e+Yn,e h is zero converges to zero for every h, so
that
dP e h dP e h dP e h
lo g n, +Yn,e == lo g n, +Yn,e _ lo n, +Yn,e o + O ( 1 )
dP dP g dP Pn,e
n,e+Yn,eho n,e n,e
TIT 1 T
==(h-h o ) ne--h lneh+-h o lneho+oP e( l).
, 2 ' 2' n,
Conclude that it suffices to show that the sequence ( n,e, ln,e) converges under 8 + Yn,e ho
to the distribution of ( , 1) under ho.
Using the general form of Le Cam' s third lemma we obtain that the limit distribution of
the sequence ( n,e, ln,e) under 8 + Yn,eh takes the form
L ( B ) == El ( 1 ) e hT e- hT Je h
h B e, e .
On noting that the distribution of ( , 1) under h == O is the same as the distribution of
( e, le), weseethatthisisequaltoEolB( , 1) dPe,h/dPe,o( , J) == Ph(( , J) E B). .
It is possible to develop a the ory of asymptotic "lower bounds" for LAMN models, much
as is do ne for LAN models in Chapter 8. Because conditionally on the ancillary statistic
1, the limit experiment is a Gaussian shift experiment, the lower bounds take the form of
mixtures of the lower bounds for the LAN case. We give only one example, leaving the
details to the reader.
9.9 Corollary. Let T,1 be an estimator sequence in a LAMN sequence of experiments
(Pn,e : 8 E 8) such that Y,;J (Tn - 1/1(8 + Yn,e h )) converges weakly under every 8 + Yn,e h
to alimit distribution Le, for every h. Then there exist probability distributions Mj (or
ratheraMarkov kernel) such that Le == EN (O, ele-l ) * M Je . In particular, COVe Le >
. 1 . T
E1/1' e le- 1/1' e .
9.6 Asymptotic Mixed Normality
133
We include two examples to give some idea of the application oflocal asymptotic mixed
normality. In both examples the sequence of models is LAMN rather than LAN due to an
explosive growth of inforrnation, occurring at certain supercritical parameter values. The
second derivative of the log likelihood, the information, remains random. In both examples
there is also (approximate) Gaussianity present in every single observation. This appears
to be typical, unlike the situation with LAN, in which the normality results from sums over
(approximately) independent observations. In explosive model s of this type the likelihood is
dominated by a few observations, and normality cannot be brought in through (martingale)
central limit theorems.
9.10 Example (Branching processes). In a Galton-Watson branching process the "nth
generation" is formed by replacing each element of the (n - l)-th generation by arandom
number of elements, independently from the rest of the population and from the preceding
generations. This random number is distributed according to a fixed distribution, called the
offspring distribution. Thus, conditionally on the size X n - 1 of the (n - l)th generation the
size X n of the nth generation is distributed as the sum of Xn-1 i.i.d. copies of an offspring
variable Z. Suppose that X o == 1, that we observe (XI, . . . , X n), and that the offspring
distribution is known to belong to an exponential family of the form
Pe(Z == z) == az eZc(e), z == 0,1,2, ...,
for given numbers ao, al, . . . . The natural parameter space is the set of all e such that
c(e)-l == L z aze z is finite (an interval). We shalI concentrate on parameters in the interior
of the natural parameter space such that {L(e) :== EeZ > 1. Set (Y2(e) == vare Z.
The sequence XI, X 2, . . . is a Markov chain with transition density
x times
pe(Y Ix) == Pe(Xn == yI X n - 1 == x) == eYC(e)x.
To obtain a two-term Taylor expansion of the log likelihood ratios, let le (y Ix) be the log
transition density, and calculate that
le(y Ix) = y - 1L(e) ,
.. y - X{L(e) x/L(e)
le (y Ix) == - -.
e 2 e
(The fact that the score function of the model e 1---+ p e (Z == z) has derivative zero yields
the identity {L(e) == -e(cjc)(e), as is usual for exponential families.) Thus, the Fisher
information in the observation (XI, ..., X n ) equals (note that Ee(X i I Xi-I) == X i - 1 {L(e))
n .. n /L(e)
-Ee Lle(X i I Xi-I) == Ee LXi-1-
. 1 . l e
J= J=
/L(e) i-I /L(e) {L(e)n - 1
== 8 {L(e) == 8 (e) - 1 .
J=l {L
For {L(e) > 1, this converges to infinity at amuch fasterrate than "usually." Because the total
information in (X 1, . . . , X n ) is of the same order as the information in the last observation
X n , the model is "explosive" in terms of growth of information. The calculation suggests
the rescaling rate Yn,e == {L(e)-n j 2, which is roughly the inverse root of the information.
134
Limits of Experiments
A Taylor expansion of the log likelihood ratio yields the existence of a point en between
e and e + Yn,e h such that
n
log Il Pe+Yn,e h (X j I X j - 1 )
j=1 pe
h n. 1 h 2 n..
== n/2 Lf e (X j IX j - 1 )+- n Lfen(XjIXj-l).
JL(e) j=1 2 JL(e) j=1
This motivates the detinitions
1 L n Xj - JL(e)X j - 1
-
n,e - ( e ) n/2. ()
JL J=1
J == 1 [ Xj - JL(e)X j - 1 X j - 1 (L(()) ]
n,e ( e ) n e 2 + () .
JL J=1
BecauseEe(Xn I X n - 1 ,..., XI) == X n - 1 JL(()), thesequenceofrandomvariablesJL(e)-nX n
is a martingale under e. Some algebra shows that its second moments are bounded as
n 00. Thus, by a martingale convergence theorem (e.g., Theorem 10.5.4 of [42]), there
exists arandom variable V such that JL(e)-n X n Valmost surely. By the Toeplitz lemma
(Problem 9.6) and again some algebra, we obtain that, almost surely under e,
1 n JL(e)
'"'X. V
JL(())n f=; } JL(e) - 1 '
1 n 1
'"'X J '-1 V.
JL(e)n f=; JL(e) - 1
It follows that the point ()n in the expansion of the log likelihood can be replaced by e at the
cost of adding a term that converges to zero in probability under e. Furthermore,
(L(e) V
Jn,e 1--* ( ) ,
() JL(e) - 1
Pe-almost surely.
It remain s to derive the limit distribution of the sequence n,e. Ifwe write X j == L 11 Zj,i
for independent copies Zj,i of the offspring variable Z, then
1 n X j-l 1 V n
b.n,e = e (e)n j 2 L L (Zj,i - /L(e)) = e (e)n j 2 L (Zi - /L(e)),
JL J=1 l=1 JL l=1
for independent copies Zi of Z and V n == L7=1 X j-l. Even though ZI, Z2, . . . and the
total number V n of variables in the sum are dependent, a central limit theorem applies
to the right side: conditionally on the event {V > O} (on which V n (0), the sequence
v,-;1/2 L :1 (Zi - JL(e)) converges in distribution to (5 (e) times a standard normal variable
G. Furthermore, if we detine G independent of V, conditionally on {V > O},t
( (5(e) IV /L(()) )
(b.n,e, Jn,e)-v-+ e V G, e(/L(e) _ 1) V .
(9.11 )
t See the appendix of [81] or, e.g., Theorem 3.5.1 and its proof in [146].
9.6 Asymptotic Mixed Normality
135
1t is well known that the event {V == O} coincides with the event {lim X n == O} of extinction
of the population. (This oceurs with positive probability if and only if ao > O.) Thus, on the
set {V == O} the series "L 1 X j converges almost surely, whence /).n,e O. 1nterpreting
zero as the product of a standard normal variable and zero, we see that again (9.11) is valid.
Thus the sequence ( n,e, Jn,e) converges also unconditionally to this limit. Finally, note
that a 2 (e) je == /L(e), so that the limit distribution has the right form.
The rnaxirnum likelihood estimator for M(e) can be shown to be asymptotically efficient,
(see, e.g., [29] or [81]). O
9.12 Example (Gaussian AR). The canonical example of an LAMN sequence of exper-
iments is obtained from an explosive autoregressive process of order one with Gaussian
innovations. (The Gaussianity is essential.) Let le I > 1 and El, E2, . . . be an i.i.d. sequence
of standard normal variables independent of a fixed variable Xo. We observe the vector
(X o , X 1, . . . , X n ) generated by the recursive formula Xt == e X t - 1 + Et.
The observations form a Markov chain with transition density p(. I Xt-1) equal to the
N (eXt-1, l)-density. Therefore, the log likelihood ratio process takes the forrn
n I n
Pn,e+Yn,e h 2 2 2
log (Xo, . . ., X n ) == h Yn,e (Xt - eX t - 1 )X t - 1 - 2 h Yn,e X t - 1 .
Pn,e t=l t=l
This has already the appropriate quadratic structure. To establish LAMN, it suffices to find
the right rescaling rate and to establish the joint convergence of the linear and the quadratic
term. The rescaling rate may be chosen proportional to the Fisher information and is taken
e -n
Yn,e == .
By repeated application of the defining autoregressive relationship, we see that
t 00
e-tx t == Xo + Le-jEj V:== Xo + Le-jEj,
j=l j=l
almost surely as well as in second mean. Given the variable Xo, the limit is normally
distributed with mean Xo and variance (e 2 - 1)-1. An application of the Toeplitz lemma
(Problem 9.6) yields
1 n V 2
_ X2
e 2n t -1 () 2 - 1 .
t=l
The linear term in the quadratic representation of the log likelihood can (under ()) be
rewritten as ()-n "L;=1 E t X t - 1 , and satisfies, by the Cauchy-Schwarz inequality and the
Toeplitz lemma,
1 n 1 n 1 n
E en 8tXt-l - en 8teHV < lein lelt-l(Ece-t+lxH - Vn 1 / 2 --+ o.
1t follows that the sequence of vectors (/).n,e, Jn,e) has the same limit distribution as the
sequence ofvectors (e- n "L;=1 Et()t-1 V, V 2 j(()2 -1)). Forevery n the vector (()-n "L;=1 Et
136
Limits of Experiments
8 t - 1 , V) possesses, conditionally on Xo, a bivariate-normal distribution. As n --+ CX) these
distributions converge to a bivariate-normal distribution with mean (O, Xo) and covariance
matrix 11(8 2 - 1). Conclude that the sequence ( n,e, Jn,e) converges in distribution as
required by the LAMN criterion. O
9.7 Heuristics
The asymptotic representation theorem, Theorem 9.3, shows that every sequence of statistics
in a converging sequence of experiments is matched by a statistic in the limit experiment.
It is remarkable that this is true under the present definition of convergence of experiments,
which involves only marginal convergence and is very weak.
Under appropriate stronger forms of convergence more can be said about the nature of
the matching procedure in the limit experiment. For instance, a sequence of maximum like-
lihood estimators converges to the maximum likelihood estimator in the limit experiment,
or a sequence of likelihood ratio statistics converges to the likelihood ratio statistic in the
limit experiment. We do not introduce such stronger convergence concepts in this section
but only note the potential of this argument as a heuristic principle. See section 5.9 for
rigorous results.
For the maximum likelihood estimator the heuristic argument takes the following form.
If h n maximizes the likelihood h r--+ dPn h, then it also maximizes the likelihood ratio
,
process h r--+ dPn,hl dPn,ho' The latter sequence of processes converges (marginally) in
distribution to the likelihood ratio process h r--+ dPh 1 dPho of the limit experiment. It is
reasonable to expect that the maximizer h n converges in distribution to the maximizer of
the process h r--+ dPhldP ho ' which is the maximum likelihood estimator for h in the limit
experiment. (Assume that this exists and is unique.) If the converging experiments are
the local experiments corresponding to a given sequence of experiments with a parameter
8, then the argument suggests that the sequence of local maximum likelihood estimators
h n == r n (en - 8) converges, under 8, in distribution to the maximum likelihood estimator
in the locallimit experiment, under h == O.
Besides yielding the limit distribution of the maximum likelihood estimator, the argu-
ment also shows to what extent the estimator is asymptotically efficient. It is efficient, or
inefficient, in the same sense as the maximum likelihood estimator is efficient or ineffi-
cient in the limit experiment. That maximum likelihood estimators are often asymptotically
efficient is a consequence of the fact that often the limit experiment is Gaussian and the
maximum likelihood estimator of a Gaussian location parameter is optimal in a certain
sense. If the limit experiment is not Gaussian, there is no a pri ori reason to expect that the
maximum likelihood estimators are asymptotically efficient.
A variety of examples shows that the conclusions of the preceding heuristic arguments are
often but not universally valid. The reason for failures is that the convergence of experiments
is not well suited to allow claims about maximum likelihood estimators. Such claims require
stronger forms of convergence than marginal convergence only.
For the case of experiments consisting of arandom sample from a smooth parametric
model, the argument is made precise in section 7.4. Next to the convergence of experiments,
it is required only that the maximum likelihood estimator is consistent and that the log density
is locally Lipschitz in the parameter. The preceding heuristic argument also extends to the
other examples of convergence to limit experiments considered in this chapter. For instance,
the maximum likelihood estimator based on a sample from the uniform distribution on [O, 8J
Problems
137
is asymptotically inefficient, because it corresponds to the estimator Z for h (the maximum
likelihood estimator) in the exponentiallimit experiment. The latter is biased upwards and
inefficient for every of the usualloss functions.
Notes
This chapter presents a few examples from a large body of theory. The notion of alimit
experiment was introduced by Le Cam in [95J. He defined convergence of experiments
through convergence of all finite subexperiments relative to his deficiency distance, rather
than through convergence of the likelihood ratio processes. This deficiency distance in-
troduces a "strong topology" next to the "weak topology" corresponding to convergence
of experiments. For experiments with a finite parameter set, the two topologies coincide.
There are many general results that can help to prove the convergence of experiments and
to find the limits (also in the examples discussed in this chapter). See [82J, [89J, [96J, [97J,
[IISJ, [138J, [142J and [144J for more information and more examples. For nonlocal ap-
proximations in the strong topology see, for example, [96J or [11 OJ.
PROBLEMS
1. Let XI, ... , X n be an i.i.d. sample from the normal N (hj,.jn, 1) distribution, in which h E .
The corresponding sequence of experiments converges to a normal experiment by the general
resu1ts. Can you see this directly?
2. If the nth experiment corresponds to the observation of a sample of size n from the uniform
[O, 1- h j n], then the limit experiment corresponds to observation of a shifted exponential variable
Z. The sequences -n(X(n) - 1) and ,.jn(2 X n - 1) both converge in distribution under every h.
According to the representation theorem their sets of limit distributions are the distributions of
randomized statistics based on Z. Find these randomized statistics explicitly. Any implications
regarding the quality of X (n) and X n as estimators?
3. Let the nth experiment consist of one observation from the binomial distribution with parameters
n and success probability h j n with O < h < 1 unknown. Show that this sequence of experiments
converges to the experiment consisting of observing a Poisson variable with mean h.
4. Let the nth experiment consists of observing an i.i.d. sample of size n from the uniform
[-1 - h / n, 1 + h / n] distribution. Find the limit experiment.
5. Prove the asymptotic representation theorem for the case in which the nth experiment corresponds
to an i.i.d. sample from the uniform [O, e - h / n] distribution with h > O by mimicking the pro of
of this theorem for the locally asymptotically normal case.
6. (Toeplitz lemma.) If an is a sequence of nonnegative numbers with I: an == 00 and X n -+ X an
arbitrary converging sequence of numbers, then the sequence I: J =I a iX i /I: J =l ai converges to
X as well. Show this.
7. Derive alimit experiment in the case of Galton- Watson branching with /L(e) < 1.
8. Derive alimit experiment in the case of a Gaussian AR( 1) process with e == 1.
9. Derive alimit experiment for sampling from a U [a, T] distribution with both endpoints unknown.
10. In the case of sampling from the U[O, e] distribution show that the maximum likelihood estimator
for e converges to the maximum likelihood estimator in the limit experiment. Why is the latter
not a good estimator?
11. Formulate and prove a local asymptotic minimax theorem for estimating e from a sample from
a U[O, e] distribution, using f(x) == x 2 as loss function.
10
Bayes Procedures
In this chapter Bayes estimators are studied from a frequentist perspec-
tive. Both posterior measures and Bayes point estimators in smooth
parametric models are shown to be asymptotically normal.
10.1 Introduction
-+
In Bayesian terminology the distribution Pn,e of an observation X n under a parameter e
-+ -
is viewed as the conditional law of X n given that arandom variable en is equal to e.
The distribution II of the "random parameter" en is called the prior distribution, and the
- -+ -
conditional distribution of en given X n is the posterior distribution. If en possesses a
density Jr and Pn,e admits a density Pn,e (relative to given dominating measures), then the
density of the posterior distribution is given by Bayes' formula
Pn,e (x) Jr (e)
P en I xn=x(e) = f Pn,e(x) dIT(e) '
This expression may define a probability density even if Jr is not a probability density itself.
A prior distribution with infinite mass is called improper.
The calculation of the posterior measure can be considered the ultimate aim of a B ayesian
analysis. Alternatively, one may wish to obtain a "point estimator" for the parameter e,
- -+
using the posterior distribution. The posterior mean E(e n I X n ) == Je P e n I X n (e) de is
often used for this purpose, but other location estimators are also reasonable.
A choice of point estimator may be motivated by a loss function. The Bayes risk of an
estimator Tn relative to the loss function -€ and prior measure II is defined as
f Ee£(T n - e) dIT(e) = E£(T n - e n)'
Here the expectation Ee-€(Tn - e) is the risk function of Tn in the usual set-up and is identical
to the conditional risk E( -€(T n - en ) I en == e) in the Bayesian notation. The corresponding
Bayes estimator is the estimator Tn that minimizes the Bayes risk. Because the Bayes risk
can be written in the form EE( -€(T n - e n) I .Kn), the value Tn == Tn (x) minimizes, for every
fixed x, the "posterior risk"
( -:- -+ _ ) _ f -€(T n - e) Pn,e (x) dD (e)
E -€(T n - en) I X n - x - f .
Pn,e (x) dD (e)
138
10.1 Introduction
139
Minimizing this expression may again be a well-defined problem even for prior densities
of infinite total mass. For the loss function l (y) == II Y 11 2 , the solution Tn is the posterior
-
mean E(8 n I X n ), for absolute loss l(y) == Ilyll, the solution is the posterior median.
Other Bayesian point estimators are the posterior mode, which reduces to the maximum
likelihood estimator in the case of a uniform prior density; or a maximum probability
estimator, such as the center of the smallest ball that contains at least posterior mass 1/2
(the "posterior shorth" in dimension one).
If the underlying experiments converge, in a suitable sense, to a Gaussian location
experiment, then all these possibilities are typically asymptotically equivalent. Consider
the case that the observation consists of arandom sample of size n from a density Pe that
depends smoothly on a Euclidean parameter e. Thus the density Pn,e has a product forrn,
and, for a given prior Lebesgue density Jr, the posterior density takes the form
(e) = fl7=1 Pe (Xi ):rr(e)
Pen IX 1 ,...,X n f fl7=1 Pe(Xi)Jr(e) de'
Typically, the distribution corresponding to this measure converges to the measure that is
degenerate at the true parameter value e o , as n --+ 00. In this sense Bayes estimators are
usually consistent. A further discussion is given in sections 10.2 and 10.4. To obtain a
more interesting limit, we rescale the parameter in the usual way and study the sequence of
posterior distributions of -Jn ( 8 n - e o ), whose densities are given by
_ h = fl7=1 Peo+h/..fi1(Xi) :rr(eo + hl-Jn)
p ..fi1(8n- e o) I Xj,... .X n () J fln (K) (e + hi -Jn) dh .
i=1 Peo+h/ vii l Jr o n
If the prior density Jr is continuous, then Jr (eo + h 1 -Jn), for large n, behaves like the constant
Jr (e o ), and Jr cancels from the expression for the posterior density. For densities Pe that
are sufficiently smooth in the parameter, the sequence of model s (Peo+h/vii: h E JR.k) is
locally asymptotically normal, as discussed in Chapter 7. This means that the likelihood
ratio processes h r-+ fl7=1 Peo+h/Viil peo (Xi) behave asymptotically as the likelihood ratio
process of the normal experiment (N (h, lia 1 ) : h E JR.k). Then we may expect the preceding
display to be asymptotically equivalent in distribution to
dN ( h, l e -I ) (X) 1
o - dN ( X 1- ) (h)
f d N ( h, 1 e 1 ) (X)d h - , eo '
where d N (JL, b) denotes the density of the normal distribution. The expression in the
preceding display is exactly the posterior density for the experiment (N (h, le-I) : h E JR.k),
_ o
relative to the (improper) Lebesgue prior distribution. The expression on the right shows
that this is a normal distribution with mean X and covariance matrix le l.
This heuristic argument leads us to expect that the posterior distribution of -Jn ( e n -
e o ) "converges" under the true parameter e o to the posterior distribution of the Gaussian
limit experiment relative to the Lebesgue prior. The latter is equal to the N (X, le I)-
distribution, for X possessing the N (O, le 1 )-distribution. The notion of convergence in this
statement is a complicated one, because a posterior distribution is a conditional, and hence
stochastic, probability measure, but there is no need to make the heuristics precise at this
point. On the other hand, the convergence should certainly include that "nice" Euclidean-
valued functional s applied to the posterior laws converge in distribution in the usual sense.
140
Bayes Procedures
Consequently, a sequence of Bayes point estimators, which can be viewed as location
functionals applied to the posterior distributions, should converge to the corresponding
Bayes point estimator in the limit experiment. Most location estimators (all reasonable
ones) map syrnrnetric distributions, such as the normal distribution, into their center of
symmetry. Then, the Bayes point estimator in the limit experiment is X, and we should
expect Bayes point estimators to converge in distribution to the random vector X, that is,
to a N (O, Ie l )-distribution under e o . In particular, they are asymptotically efficient and
asymptotically equivalent to maximum likelihood estimators (under regularity conditions).
A remarkable fact about this conclusion is that the limit distribution of a sequence of
Bayes estimators does not depend on the prior measure. Apparently, for an increasing
number of observations one's prior beliefs are erased (or corrected) by the observations. To
make this true an essential assumption is that the prior distribution possesses a density that
is smooth and positive in a neighborhood of the true value of the parameter. Without this
property the conclusion fails. For instance, in the case in which one rigorously sticks to a
fixed discrete distribution that does not charge e o , the sequence of posterior distributions of
en cannot even be consistent.
In the next sections we make the preceding heuristic argument precise. For technical
reasons we separately consider the distributional approximation of the posterior distributions
by a Gaussian one and the weak convergence of Bayes point estimators.
Even though the heuristic extends to convergence to other than Gaussian location exper-
iments, we limit ourselves in this chapter to the locally asymptotically normal case. More
precisely, we even assume that the observations are arandom sample X 1, . . . , X n from a
distribution Pe that admits a density Pe with respect to a measure j.l on a measurable space
(X, A). The parameter e is assumed to belong to a measurable subset e of IR. k that contains
the true parameter e o as an interior point, and we assume that the maps (e, x) 1-+ Pe (x) are
jointly measurable.
All theorems in this chapter are frequentist in character in that we study the posterior laws
under the assumption that the observations are arandom sample from Peo for some fixed,
nonrandom e o . The alternative, which we do not consider, would be to make probability
statements relative to the joint distribution of (X 1, . . . , X n, en), given a fixed prior marginal
- -
measure for en and with P being the conditionallaw of (X 1, . . . , X n ) given en.
10.2 Bernstein-von Mises Theorem
The heuristic argument in the preceding section indicates that posterior distributions in dif-
ferentiable parametric models converge to the Gaussian posterior distribution N (X, Iia 1 ).
The Bernstein-von Mises theorem makes this approximation rigorous and actually yields
the approximation in a stronger sense than discussed so faro In Chapter 7 it is seen that the
observation X in the limit experiment is the asymptotic analogue of the "locally sufficient"
statistics
1 -1'
f'o,. n, eD = -fii f::r. I eD feD ( X J ,
where fe is the score function of the model. The Bernstein-von Mises theorem asserts
that the total variation distance between the posterior distribution of -fii( 8 n - e o ) and the
random distribution N( n,eo' Ie l) converges to zero. Because n,eo X, this has as a
10.2 Bernstein-von Mises Theorem
141
consequence that the posterior distribution of -JfiC EJ n - e o ) converges, in any reasonable
sense, in distribution to NCX, le I).
The conditions of the following version of the Bemstein-von Mises theorem are re-
markably weak. Besides differentiability in quadratic mean of the model, it is assumed
that there exists a sequence of uniformly consistent tests for testing Ho : e == e o against
HI : II e - (jo II > £, for every £ > O. In other words, it must be possible to separate the
true value (jo from the cornplements of balls centered at e o . Because the theorem implies
that the posterior distributions eventually concentrate on balls of radii Mnl -Jfi around e o ,
for every M n 00, this separation hypothesis appears to be very reasonable. Even more
so, since, as is noted in Lemmas 10.4 and 10.6, under continuity and identifiability of the
model, separation by tests of Ho : e == (jo from HI : II (j - (jo II > £ for asingle (1 arg e ) £ > O
already implies separation for every £ > O. Furthermore, if EJ is compact and the model
continuous and identifiable, then even the separation condition is superfluous Cbecause it is
automatically satisfied). t
10.1 Theorem (Bernstein-von Mises). Let the experiment CPe : (j E 8) be differentiable
in quadratic me an at (30 with nonsingular Fisher information matrix leo, and suppose that
for every £ > O there exists a sequence of tests q;n such that
P CPn O,
sup Pen Cl - q;n) O.
II e - eo II :::: 8
C10.2)
Furthermore, let the prior measure be absolutely continuous in a neighborhood of (30 with
a continuous positive density at 0 0 . Then the corresponding posterior distributions satisfy
pn
II P - - N ( !1 1-1 )11 O
,Jn (e n - eo) I XI,..., X n n, eo ' e o .
Proof. Throughout the pro of we rescale the parameter e to the local parameter h ==
-JfiC(3 - (jo). Let ITn be the corresponding prior distribution on h Chence ITnCB) == ITC(jo +
B I -Jfi)), and for a given set C let IT be the probability measure obtained by restricting
ITn to C and next renormalizing. Write Pn,h for the distribution of X n == CX 1, . . . , X n )
under the original parameter (30 + hi -Jfi, and let Pn,c == f Pn,h dIT Ch). Finally, let
H n == -JfiC 8 n - ( 0 ), and denote the posterior distributions relative to ITn and IT by PHn I x n
and P I -> , respectively.
Hn X n
The proof consists of two steps. First, it is shown that the difference between the posterior
measures relative to the priors ITn and IT n, for C n the ball with radius M n , is asymptotically
negligible, for any M n 00. Next it is shown that the difference between N C!1n,e o ' le l)
and the posterior measures relative to the priors IT;n converges to zero in probability, for
some M n 00.
For U, a ball of fixed radius around zero, we have Pn U <1 r> Pn o, because Pn h <1 r> Pn o
" , n ,
for every bounded sequence h n , by Theorem 7.2. Thus, when showing convergence to zero
in probability, we may always exchange Pn,o and Pn,U'
t Reeall that a test is a measurable function of the observations taking values in the interval [O, 1]; in the present
eontext this means a measurable funetion 4Jn : X n f--+ [O, 1].
142
Bayes Procedures
Let en be the ball of radius M n . By writing out the conditional densities we see that, for
any measurable set B,
PHnIXn(B) - p nlXn (B) = PHnIXJc n B) - PHnIXJc ) P :IXn (B).
Taking the supremum over B yields the bound
IIPHn'Xn - p nlxJ < 2PHnIXJC ).
The right side will be shown to converge to zero in mean under Pn,u for U a ball of fixed
radius around zero. First, by assumption and because Pn,u <1 Pn,o,
Pn,U PHn I in (e ) == Pn,u PHn I in (e ) (1 - <l>n) + 0(1).
Manipulating again the expressions for the posterior densities, we can rewrite the first term
on the right as
iln(e ) 1 1
p c - -. U 1 - < P 1 - df1 h
I1n (U) n,C n Hn I Kn ()( <Pn) - I1n (U) ce n,h ( <Pn) n ( ).
n
For the tests given in the statement of the theorem, the integrand on the right converges
to zero pointwise, but this is not enough. By Lemma 10.3, there automatically exist tests
<l>n for which the convergence is exponentially fast. For the tests given by the lemma the
preceding display is bounded above by
1 [ e- c (lI h I1 2 /\n) dI1n (h).
iln(U) Jllhll Mn
Here f1 n (U) == il (e o + U / -Jn) is bounded below by a term of the order 1/ -Jnk, by the
positivity and continuity of the density n at go. Splitting the integral into the domains
M n < Ilhll < D-Jn and Ilhll > D-Jn for D < 1 sufficiently small that n(g) is uniformly
bounded on Ilg - gall < D, we see that the expression is bounded above by a multiple of
[ e-cllhll2 dh + ,Jnk e- cD2n .
Jllhll Mn
This converges to zero as n, M n -+ 00.
In the second part of the proof, let e be the ball of fixed radius Maround zero, and let
N C (f.L, b) be the normal distribution restricted and renormalized to e. The total variation
distance between two arbitrary probability measures P and Q can be expressed in the form
IIP - QII == 2 J(l - p/q)+ dQ. It follows that
i 1\ N C (L\n,e o , le;; 1 ) - P£ I in II
J ( d N C ( L\ n, Đ o ' [e; 1 ) (h ) ) + C
== 1 - dP- -. (h)
1 c (h)Pn,h(X n )nn(h)/ Je Pn,g(Xn)nn(g) dg Hn IX n
< If( l - Pn,g cXn)Jr n (g)dN c (lin,Đ o , l l)(h) ) + dNC ( L\ [-1 )( ) dP -. ( h )
- e ( -1 ) n, e o ' Đo g H IX'
Pn,h(Xn)nn(h) dN L\n,e o , leo (g) n n
10.2 Bernstein-von Mises Theorem
143
because (1 - EY)+ < E(l - Y)+. This can be further bounded by replacing the third
occurrence of NC( n,eo' Ie l) by a multiple of the uniform measure Ac on C. By the
dominated-convergence theorem, the double integral on the right side converges to zero in
mean under Pn,c if the integrand converges to zero in probability under the measure
p n, C (d x) P-%'I I X n =x (d h) Ac (d g) == 11 (d h) P n, h (d x) Ac (d g ) .
(Note that Pn,c is the marginal distribution of X n under the Bayesian model with prior
IT;.) Here 11; is bounded up to a constant by Ac for every sufficiently large n. Because
Pn,h <I r> Pn,o for every h, the sequence of measures on the right is contiguous with respect
to the measures Ac(dh) Pn,o(dx) Ac(dg). The integrand converges to zero in probability
under the latter measure by Theorem 7.2 and the continuity of Jr at e o .
This is true for every ball C of fixed radius M and hence also for some M n ---* 00. .
10.3 Lemma. Under the conditions of Theorem 10.1, there exists for every M n ---* 00 a
sequence of tests CPn and a constant c > O such that, for every sufficiently large n and every
Ile - 8 0 11 > Mn/ '
P CPn ---* O,
P;(l - CPn) < e- cn (lIe- e oIl 2 /d).
Proof. We shall construct two sequences oftests, which "work" for the ranges M n / <
II e - 8 0 II < s and II e - e o II > s, respectively, and a given s > O. Then the CPn of the lemma
can be defined as the maximum of the two sequences.
First consider the range M n / < II e - e o II < s. Let it be the score function
truncated (coordinatewise) to the interval [ - L, L]. By the dominated convergence theorem,
Peoi oi ---* leo as L ---* 00. Hence, there exists L > O such that the matrix Peoiti is
nonsingular. Fix such an L and define
UJ n = 1 { II (JP'n - Peo)i o II > J Mn/n }.
By the central limit theorem, P% W n ---* O, so that W n satisfies the first requirement. By the
triangle inequality,
II (P n - Pe )i o II > II (Peo - Pe )i o II - II (P n - Peo )i II.
Because, by the differentiability of the model, Pei o - Peoi o == (Peoit i + 0(1)) (e - e o ),
the first term on the right is bounded below by c II e - e o II for some c > O, for every e that is
sufficiently close to e o , say for II e - e o II < s. If W n == O, then the second term (with out the
minus sign) is bounded above by ,J Mn/n. Consequently, for every clle - e o II > 2 ,J M n / n,
and hence for every Ile - e o II > M n / and every sufficiently large n,
P;(1 - UJ n ) < Pe (II (JP'n - Pe)i o II > clle - eoll) < e-Cnlle eoIl2,
by Hoeffding's inequality (e.g., Appendix B in [117]), for a sufficiently small constant C.
Next, consider the range Ile - e o II > s for an arbitrary fixed s > O. By assumption there
exist tests CPn such that
Pe: CPn ---* O,
sup P; (1 - CPn) ---* O.
lIe-eo 11>8
144
Bayes Procedures
It suffices to show that these tests can be replaced, if necessary, by tests for which the
convergence to zero is exponentially fast. Fix k large enough such that P CPk and P: (1- CPk)
are smaller than 1/4 for every Ile - eoll > 8. Let n == mk + r for O < r < k, and define
Y n ,l, . . . , Yn,m as CPk applied in tum to XI, . . . , X k , to X k + 1 , . . . , X 2k , and so forth. Let Y n,m
be their average and then define W n == 1 {Y n,m > 1/2}. Because EeYn,j > 3/4 for every
Ile - e o II > 8 and every j, Hoeffding's inequality implies that
pn ( 1 - W ) == "P, e ( Y < 1 / 2 ) < e-2m( - )2 < e- m / 8 .
e n n,m - -
Because m is proportional to n, this gives the desired exponential decay. Because Eeo Yn,j <
1/4, the expectations P W n are similarly bounded. .
The Bemstein-von Mises theorem is sometimes written with a different "centering se-
quence." By Theorem 8.14 any sequence of standardized asymptotically efficient estimators
-Jii(e n - e) is asymptotically equivalent in probability to n,e. Because the total variation
distance
IIN( n,e, le-I) - N(-Jii(e n - e), le-I) II
is bounded by a multiple of II n,e - -Jii(e n - e) II, any such sequence -Jii(e n - e) may
replace n,e in the Bemstein-von Mises theorem. By the invariance of the total variation
norm under location and scale changes, the resulting statement can be written
( Al -1 ) Po
P e n I Xl,...,X n - N en, n le -+ O.
Under regularity conditions this is true for the maximum likelihood estimators en. Com-
bining this with Theorem 5.39 we then have, informally,
( A 1 -1 )
Pen len N en, n len
and
( - 1 -1 )
Pen I e n N en, n len '
since conditioning en on e n == e gives the usual "frequentist" distribution of en under e.
This gives a remarkable symmetry.
Le Cam' s version of the Bemstein-von Mises theorem requires the existence of tests
that are uniformly consistent for testing Ho : e == e o versus HI : II e - e o II > 8, for every
8 > O. Such tests certainly exist if there exist estimators Tn that are uniformly consistent,
in that, for every 8 > O,
supPe(IITn -eli > 8)-+0.
e
In that case, we can define CPn == 1 { II Tn - e o II > 8/2}. Thus the condition of the Bernstein-
von Mises theorem that certain tests exist can be replaced by the condition that uniformly
consistent estimators exist. This is often the case. For instance, the next lemma shows that
this is the case for a Euclidean sample space X provided, for Fe the distribution functions
corresponding to the Pe,
inf II Fe - Fe' II 00 > O.
lIe-e'II>8
10.2 Bernstein-von Mises Theorem
145
For compact parameter sets, this is implied by identifiability and continuity of the maps e
Fe. We generalize and formalize this in a second lemma, which shows that uniformity on
compact subsets is always achievable ifthe model (Pe : e E 8) is differentiable in quadratic
mean at every e and the parameter e is identifiable.
A class of measurable functions F is a uniform Glivenko-Cantelli class (in probability)
if, for every £ > O,
sup P p (11JID n - p IIF > £) -+ o.
p
Here the supremum is taken over all probability measures P on the sample space, and
II Q II F == sup f EF I Qf I. An example is the collection of indicators of all cells (- 00, tJ in a
Euclidean sample space.
10.4 Lemma. Suppose that there exists a uniform Glivenko-Cantelli class F such that,for
every £ > O,
inf II Pe - Pe,llF > O.
d(e,()'»B
(10.5)
Then there exists a sequence of estimators that is uniformly consistent on 8 for estimat-
ing e.
10.6 Lemma. Suppose that 8 is a -compact, Pe =1= P e , for every pair e =1= e', and the maps
e t-+ Pe are continuous for the total variation norme Then there exists a sequence of
estimators that is uniformly consistent on every compact subset of B.
Proof. For the proof of the first lemma, define en to be a point of (ne ar) minimum of
the map e r-+ II]P n - Pe II F. Then, by the triangle inequa1ity and the definition of en,
IIPen - Pell.r < 211JP n - Pell F + lin, ifthe near minimum is chosen within distance lin
of the true infimum. Fix £ > O, and let 8 be the positive number given in condition (10.5).
Then
Po (dCB n , e) > e) < Po (II POn - Po IIF > 8) < Po (2 11 JP' n - Po IIF > 8 - ).
By assumption, the right side converges to zero uniformly in e.
For the proof of the second lemma, first assume that 8 is compact. Then there exists
a uniforrn Glivenko-Cantelli class that satisfies the condition of the first lemma. To see
this, first find a sequence Al, Az, . .. ofmeasurable sets that separates the points Pe. Thus,
for every pair e, e' E 8, if Pe (Ai) == P e , (Ai) for every i, then e == e'. A separating
collection exists by the identifiability of the parameter, and it can be taken to be countable
by the continuity of the maps e r-+ Pe. (For a Euclidean sample space, we can use the
cells (-00, tJ for t ranging over the vectors with rational coordinates. More generally,
see the lemma below.) Let F be the collection of functions x r-+ i -11 Ai (x). Then the
map h: 8 r-+ fOO(F) given by e r-+ (Pef)fEF is continuous and one-to-one. By the
compactness of B, the inverse h- 1 : heB) r-+ 8 is automatically uniformly continuous.
Thus, for every 8 > O there exists 8 > O such that
II h ( e) - h (e') II F < 8 imp lies d ( e, e') < £.
146
Hayes Proadurt!s
This mcans that (10.5) is satisfied. The class :F is also a unifonn Glivenko-Cantclli class,
because by Chebyshev's inequality,
Pr(UP n - PIrF . > &) '" Pp(lPnl - PIl> E) " ..
L.- L...J ne",,,
I t
This concludes the proof of the second Jemma for compact 8.
To rcmovc the compactncss condition. wrile e as the union of an incrcasing scqucncc
of compact scts K, C Kz C .... f"Or cvcry 111 rherc exists a sequcnce of estimators .
Tri,,,, that is uniformly consistcnt on Km' by the preceding argument. Thus, for c\'cry
fixed m, .
a lt . m : = sup Po ( d(T".m, O) 2: .!. ) -. O.
OcK In
11 -. 00.
Then therc exists a scqucncc IIl'J -. 00 such that a,t.m" O as" 00. It is not hard to sce
that On = Tn.m" satistics the requiremcnts _
As a co.nscqucnce of the second lemma, if there. cxists a scqucnce of tcsts t/Jn such that
(10.2) holds for same t: > O, then it holds for e\'ery e > O. Jn that ca we.can replace the
gi\'cn sequcnce r/Jn by the minimum of 4>" and the tcsts I {nT,. - 0 0 11 ::: &/2 J for a scqucnce
of estimators 'f" that is unifonnly consistcnt on a sufticicnrJy Jarge subsct of B.
10.7 [.,emma. Le' ,he .fel olprobabililY mt?il.wre.'i P (JIl ti u1('aJurabh' .'i/wce (,l'. A) !Je
.'i{'parablc for tlu.' ((Jta! "ilrial;on "orm. The" (here f!.\'isIs (I cOlllltable .wbsel Ao c A .mch
1/1(11 PI =. P2 011 .Ao implies Pa = 1'2 for e\'el)' PI, P!. E P.
Proo/. Thc set P can bc identificd with a subsct of LI ('l) for 3 suitable probability m asurc
Il. For instance. II. c:m he laken a con\'cx linear combination of a countablc dense sct. Let
Po be a count.able densc.subsct, and Jet .Ao he the set of all finirc intcrscctions of Ihe scts
p-I (B) for p ronging ovcr a choke of dcnsiticsof the set Po c Lt (/.1.) :md B ranging over
a countahle generator of the Borel sets in JR.
l1\c n e\'ery density p E Po is O'(Ao)-measumble by construction. A densit)' of a mcas\lrc
p E P - Po can bc approximatcd in LI (/l) by a scquencc from Po and hence can he chosen
(1 (.Ao)-measurable. without loss of gencmlity.
Becausc Ao is intersection-stablc (a "n-system"). two probability mcasurcs th 1I :Jgrec
on AoautomaticaJly agrcc on the a-fleld (1 (A o ) gencrated by Ao. Then they also gi\'c the
same expectation to cvery a (Ao)-mcasurable funclion f: X..... [O. II. If the measures
have u (.A o )-mcasurable densities. then they must agrce on.A. becausc P(t\) = E,I IAP =
E'I E,. (I Ala (..4 0 ») p ir p is (7 (.A o )-measumble. _
10.3 Point Eslimators
The Bemstcin-von M iscs theorcm sh()ws that the posterior laws con\'erge in distribution to a
Gaussian posterior law in total variation distance. As a conscquence, any location functional
that is suitably continuous rcl..iive to the total \' iriation norm .1pplied lO the scqucncc of
10.3 Po;1I1 Es,;mators
147
postcrior laws convergcs to thc.same location functionalapplied to the limiting G"ussian
postcri(U distribution. For most choiccs this mcans to X, or a. N (O. ./;" )-distribution.
In this section we consider more general Bayes point estimators that arc defined as the
minimiz.ers of the posterior risk functions relati\'e to some loss function. For a given loss
function l : R.t H> [0.00). let Tn, forfixed XI. .... X 'I' minimi7.c the postcrior risk
J €( (I - O» n;'"" !,,,(X,)dn(O)
I H> J n " .
(_I !'tJ(X,) dn (O)
IL is not immediately c1ear that the minimi7.ing values T" can be sc1ccted as a measurable
function of (he obscrvations. This is an implicit assumption, or othcrwise the statemcnts
are to be undcrsto(Jd rdath' lo outcr probabilitic . \Vc also makc it an implicit assumption
that the integrals in the prcccding display cxist, for al most c\'ery scqucncc of obscrvations.
To dcri\'c the limit distribution of (T n - 0 0 ), \vc apply general rcsuhs on Al-cstimators.
in particular the argmax continuous-mapping lhcorcm, Theorem 5.56.
Wc rcstricI oursclvcs to loss functions with the propcrty, for c\'cry AI > O,
sup t(h)::; inf ((Il).
'hn::'" Ihll 2M
with strict inequality for at leasIone Al.f 111is is true, for instanct:. for los..c; functions of
the form l(h) = to(lIh n) for a nondccrcasing function to: [O. 00) H> [0,00) that is not
constant on (O. 00). Funhcnnorc, we suppose that t. grows at most polynomially: For somc
constant p O.
t(h) S I + nit II".
10.8 TJzeorcm. ut Ilte cOlldirioll. ofTheol"em 10.1 iroM. wul Jet l smis/)' the cOllclitio1l,fi cu
/istt'd,lor a p .mcll tltal f lIOJI" cJn(O> < 00. The" lire .f(>CJIIt!llce ..;'ii(T N - ( 0 ) com'ery:e.f
111lderO() ill distriblllioll to Ihe millimi:.erofl f ((I-Il) d N(X, If I ) (1I).for X possc.r:;;ng
the N (O. I I )-cJi,ftriblltioll, prm'Med tira I C/ll." IwO minimi:.er... oflhis proccs ' c:oilldde Cl/mo.rt
.'illri.'/y. I" purric"lar, for e\'ery fWIl:.ero. .wbcoII\'('.'( lOJ," fmtt.tima il (.'on\'('rges 10 X.
.Proo/. We adopt the notation as Ii ted in th first paragraph of the proof ofThcorcm 10.1.
The last asscrtion of the theorem is a conscquence of Andcrson'slcrnma. Lcmma 8.5.
The standardi7.cd estimator (T" - 00) minimizcs the funclion
Z ( f l(l - It> Pn.II(X,,) c1n,,(h)
1 ." t) = f - . = Fti IX eh
P",/r(X,,)dnn(/l) · II
whcre li is the function II t- l(1 - Ir). ll1C proof consists of thrcc parts. First it is shown
that integrals over the sets 1I1t \I M" can be neglected for evcr)' M" 00. Next, it is proved
that the sequence (7" - 00) is unifonnly tight. Finally. it is shown that the stoc:hastic
pmccsses 1 Z,,(I) converge in distribution in the spacc [""(K). forc\'cry compact K. to
the process
I Z(l) = f (I - h)c/N(X.I, I)(,,).
t Th..: 2 is for convcnicncc. any other numhcr would do.
148
Bayes Procedures
The sample paths of this limit proeess are eontinuous in t, in view of the subexponential
growth of -€ and the smoothness of the normal density. Henee the theorem follows from the
argmax theorem, Corollary 5.58.
Let C n be the ball of radius M n for a given, arbitrary sequenee M n -+ 00. We first show
that, for every measurable funetion f that grows subpolynomially of order p,
Pn o
Pjjn I in (flc ) o.
(10.9)
To see this, we utilize the tests CPn for testing Ho : e == e o that exist by assumption. In view
of Lemma 10.3, these may be assumed without loss of generality to satisfy the stronger
property as given in the statement of this lemma. Furthermore, they can be constructed
to be nonrandomized (i.e., to have range {O, I}). Then it is immediate that (Pjjn I in f)CPn
converges to zero in Pn,o-probability for every measurable function f. Next, by writing out
the posterior densities, we see that, for U a fixed ball around the origin,
Pn U R H - I x .... ( flce ) (1 - 4Jn) = 1 1 f(h)Pn h [ R H - I x .... (U)(1 - CPn) ] dITn (h)
, n 11 n ITn (U) ce ' n n
n
< 1 1 (1 + IIhIlP)e-c(llhI12/\n) dITn(h).
ITn (U) ce
n
Here ITn (U) is bounded below by a term of the order 1/ -Jiik, by the positivity and continuity
at e o of the prior density Jr. Split the integral over the domains M n < Ilh II < D-Jii and
Ilh II > D-Jii, and use the fact that f Ile IIP dIT (e) < 00 to bound the right side of the display
by terms of the order e-AM and -Jiik+p e- Bn , for some A, B > O. These converge to zero,
whence (10.9) has been proved.
Define -€(M) as the supremum of -€(h) over the ball of radius M, and :f(M) as the
infimum over the complement of this ball. By assumption, there exists 8 > O such that
rJ: == 1::(28) - -€ (8) > O. Let U be the ball of radius 8 around O. For every II t II > 3M n and
sufficient1y large M n , we have -€(t - h) - -€( -h) > rJ if h EU, and f(t - h) - -€( -h) >
:f(2M n ) --€(M n ) > O if h E ue n Cn, by assumption. Therefore,
Zn(t) - Zn(O) = PHnIXn[U (t -h) -l(-h))(lu + 1u c nc n + 1C )]
> rJPjjn I in (U) - Pjjn I in (-€( -h)lc ).
Here the posterior probability Pjjn I in (U) of U converges in distribution to N(X, Ie l )(U),
by the Bemstein-von Mises theorem. This limit is positive almost surely. The second
term in the preceding display converge to zero in probability by (10.9). Conclude that the
infimum of Zn (t) - Zn (O) over the set of t with Ilt II > 3M n is bounded below by variables
that converge in distribution to a strict1y positive variable. Thus this infimum is positive
with probability tending to one. This implies that the probability that t r-+ Zn (t) has a
minimizer in the set Iltll > 3M n converges to zero. Because this is true for any M n -+ 00,
it follows that the sequence -Jii (Tn - e o ) is uniformly tight.
Let C be the ball of fixed radius Maround O, and fix some compact set K C k. Define
stochastic processes
Z n, M (t) == Pjj n I i n (-€ t 1 c ) ,
Wn,M == N ( n,eo' Ie l) (ft 1 c ),
W M == N(X, I l)(-€tlc).
10.4 Consistency
149
The function h f (t - h) 1 c (h) is bounded, uniforml y if t ranges over the cornpact K.
Renee, by the Bemstein-von Mises theorern, Zn,M - Wn,M O in fOO(K) as n --* 00,
for every fixed M. Second, by the continuous-rnapping theorern, Wn,M W M in fOO(K),
as n --* 00, for fixed M. Next W M Z in foo (K) as M --* 00, or equivalently C t }Rk.
Conclude that there exists a sequence M n --* 00 such that the processes Zn,M n Z in foo (K).
Because, by (10.9), Zn (t) - Zn,M n (t) O, we finally conclude that Zn 'v'7 Z in fOO(K). .
*10.4 Consistency
A sequence of posterior measures P 8 n I X1,...,X n is called consistent under e if under P -
probability it converges in distribution to the measure 8e that is degenerate at e, in proba-
bility; it is strongly consistent if this happens for almost every sequence XI, X 2, . . . .
Given that, usually, ordinarily consistent point estirnators of e exist, consistency of
posterior meas ures is amodest requirement. If we could know e with almost complete
accuracy as n --* 00, then we would use a Bayes estimator only if this would also yield the
true value with similar accuracy. Fortunately, posterior meas ures are usually consistent.
The following famous theorem by Doob shows that under hardly any conditions we already
have consistency under almost every parameter.
Recall that 8 is assumed to be Euclidean and the rnaps e Pe (A) to be measurable
for every measurable set A.
10.10 Theorem (Doob's consistency theorem). Suppose that the sample space (;\:', A) is
a subset of Euclidean space with its Borel a-field. Suppose that Pe =f=. Pet whenever e =f=. e'.
Then for every prior probability measure il on 8 the sequence of posterior measures is
consistent for rr -almost every e.
ProoJ. On an arbitrary probability space construct random vectors 8 and XI, X 2, . . .
- -
such that 8 is marginally distributed according to il and such that given 8 == e the vectors
XI, X 2 , . . . are i.i.d. according to Pe. Then the posterior distribution based on the first n
observations is P 8 1 Xl,..',X n ' Let Q be the distribution of (X 1, X 2 , . . . , 8) on ;\:'00 x 8.
The main part of the proof consists of showing that there exists a measurable function
h : :\:,00 f--* 8 with
h(XI, X2, ...) == e,
Q-a.s..
(10.11)
Suppose that this is true. Then, for any bounded, measurable function f : 8 JR, by
Doob' smartingale convergence theorern,
E(f( 8 ) I XI, ..., X n ) --* E(f( 8 ) I XI, X 2 , ...)
f(h(X I , X 2 ,.. .)), Q-a.s..
By Lemma 2.25 there exists a countable collection :F of bounded, continuous functions f
that are determining for convergence in distribution. Because the countable union of the
associated null sets on which the convergence of the preceding display fails is a null set,
we have that
P- 8 h
el Xl,...,X n (X 1 ,X2,...)'
Q-a.s. .
ISO
Ba>' s Procedures
This statcmcnt rcfers to the marginal distribution of (X.. X2.. ..) under Q. We wish to
translate it into a statcmcnt conccming the peoo-mcasures. Let C C Xoo x e be th intcr-
section of the scts on which the weak convergcnce holds and on which (15.9) is valid. By
Fubini's thcorem
1= "Q(C) = II Ic(x.8)dP (x)dn(O):= I P,., (Co)dn(O).
where et' = {x: (x, O) E c} is the horizontaJ scction of C at height O. It follows that
PIJOO(C H ) = J for n-aJmost cvcry (). For c\'cry 8 such that Pt]' (Ctl) == J. \Ve have that
(x. O) e C for P -almost e\'ery se<)uence X" X2. . .. and hencc
PHIXI .rl.....X.:..r" - đhbs..t1....1 = S, ,
This is the assertion of the thcorem.
In order to establish (15.9). c:aJ1 a mc surable function f: e Ho R l1ccess;ble if therc
cxists a sequence of measuiJbl functions lin: X H a such that
II I",,(X) -/(8)11\ 1 dQ(x, O) --i> O.
(Bere we abuse notation in vicwing hIl also aS a measurablc function on X x H.) Then
there also cxists a (sub)scquencc with hil (.t) -.. J(O) al most surely under Q. whcnce c\'cry
accessibJc function fis a]most cverywhcre equal to an A Xo x {0. 0}-mea!\urablc functiori.
This is a measurable function of:c = (XI, X . . . .) alone. Ir we can show that the functions
f (O) =" O,, are acccssible. then (15.9) foJlows. \Ve shali in fact show that cvcry Borci
measurable function is 3ccessiblc.
By the strong law of laftJc numbers. lin (X) = L !!::I I" (x,) - PH(I\) aJmost surcly unu r
P . for cvcry O and mcasur.lb]c set A. Consequemly. by th dominatcd con\'crgcncc
theorem.
II l"n(X) - Pu(A>1 dQ(:c. 9) O.
Thus each of the functions O ,(A) is accessible.
Because (..1:#. A) is Euclidean by assumption. there cxists a countable measurc-detcr-
mining subcollcction Aa c A. The functions O .... PH(A) are measur.lble byassumption
and separarc the points of (-J as A rangcs ovcr .Ao, in vic\V of the choicc of Ao and the
idemifiabililY of the parameter O. This implies that thcse functions generale the Borci
u-field on 8. in view of Lemma 10.12.
The proof is comp]ctc once it is shown that evcry function that is measurablc in the
u-fidd gencrated by the accessible functions (which is the BorcI u-field) is 3ccessibJc.
From the definirion jt follows easi1}' that the set of accessib]e functions is a v ctor .space.
contains the constant functions, is closed undcr monotone ]imits. and is a lauice. The
dcsircd result thcrefore fotlows by amonotone class argument. as in Lcmma 10. ) 3. .
111c merit of the prcceding theorem is that it imposcs haru[y any conditions. but its
drawback is that it givcs thc consistcnc}' only up to nuli scts of possiblc paramcIcrs (dc-
pcntling on the prior). In cenain ways thcse nuli set s can he quite large. and cxamplcs ha\'c
10.4 C{)n. ;.'IleIlQ'
151
bccn constructed where Bayes estimators bchavc badly. To guarantcc consistcncy undcr
c\'ery paramctcr it is necessary to imposc so mc furthcr conditions. Because in this chapter
wc are mainly conccrned with as)'mptotic nom1alit)' of Bayesestimators (which implies
consistcncy with a rate). we omit a discus....ion.
10.12 [..emma. Lel:F be cl cOlllltab/eco/l('( ,;O" ofmea.'IlImblefll/l{.ti01zS I: e c RZ: .- IR
thll1 .'i('/mrate.f lhe po;nrs 01 EJ. The" the Borel a.field and the u.Jleld generated by F 01Z e
co;ndde.
Proof. By assumption, the map": E-) ..... !RF defined by h(O)f = f(O) is mcasurable and
one-to-one. Because F is countablc, thc Borci O'-field on RF (for the product topology) is
equal to the n-field gcncratcd by the coordinate projections. Hencc the a-ticlds generated
by" and F (viewed as Borci measurabJe maps in RF and iR. respccti\'cJy) on e are idcnrical.
Now h-I, defined on the range ofh, is automatically Borci measumhlc. by Proposition 8.3.5
in (24J. and hencc(-) and h«..) arc Borci isomorphic. .
IO.t3 u/mna. Let F bt' u/illl'ar :ru/J.'ipace of £1 (n) with the pmpertie.f
(i) if f. K EF. ,hell f " g E :F:
(ii) ifO II f!. ... EF limI f,J t f E .cl (n). thell f E:F;
(iii) I E :F.
The1l :F f.'OlltCli1lS e,'er)" ('I (:F)-m(>a.wrable fil11ctiml iII .c. (n).
PriJo/' Bccausc any O'(F )-measumble nonncgative function is the monotone limit of a
scquence of simple functions. it suffices (o provc that JA € :F for every A e o' (F). Define
Ao = {A : lA E :Fl. Then An is an intersection-stablc Dynkin system and hence a l1-field.
Furthcnnore. for e\ cry f e F and a E R. the functions Il(f- a)+ 1\ I are containcd in:F
and increase pointwise to I ((>ut. It foJlows Ihal (f > al E Ao. Ilenee a{:F) c AJ' .
Notes
The Bernstein-\'on Miscs theorem has that name, bccausc. ns Le Cam and Y mg 197) wrltc. il
was first discovcred by Laplace. The theorem thott is presented in this chapter is considerably
more elcgant than the rcsults. by these early authors. and also much bcuer than the result
in Le Cam (91 J. who revi\'cd the theorcin in order to provcrcsulrs on supcrcfficiency. \Vc
adaptcd it from Le Cnrn 1961 and Lc Cam and Y;mg 1971.
Ibragimov and l-Iasminskii (80] discuss the convergcncc of Bayes point cstimators in
grcater gcneralily, and also co\'er non-Gausshm limit cxpcrimcms. but their discussion of
the i.i.d. ease as discussed in the present chapter is limited to boundcd parameter sets and
requircs stronger assumptions. Our treatmcnt uses some elements of their proof. but is
heavily bascd on Le C'1m's Bcrnstcin-\'on Miscs thcorcm. Inspcetion of the proof shows
th ,t th conditions on the loss function can bc rclaxed signilicantl)'. for instancc allowing
cxponential growth.
Doob's theorcm originatcs in (39). The JX>tcntial nuli scts of inconsistency that il leil\'cs
o n really cxist in some situations particularly il' the paramctcr set is inflnitc dimensional.
152
Bayes Procedures
and have attracted much attention. See [34], which is accompanied by eva1uations of the
phenomenon by many authors, including Bayesians.
PROBLEMS
1. Verify the conditions of the Bernstein-von Mises theorem for the experiment where Pa is the
Poisson measure of mean () .
2. Let Pe be the k-dimensional normal distribution with mean () and covariance matrix the identify.
Find the a posteriori law for the prior rr == N(T, A) and some nonsingular matrix A. Can you
see directly that the Bemstein-von Mises theorem is true in this case?
3. Let Pe be the Bernoulli distribution with mean (). Find the posterior distribution relative to the
beta-prior measure, which has density
() r--+ r ( a ) r (tJ) () a -1 (1 _ () ) f3 -11 ( () ) .
r(a + tJ) (0,1)
4. Suppose that, in the case of a one-dimensional parameter, we use the loss function l(h)
1 (-1,2) (h). Find the limit distribution of the corresponding Bayes point estimator, assuming that
the conditions of the Bemstein-von Mises theorem hold.
11
Projections
A projection of arandom variable is de.fined as a closest element in a
given set of functions. We can use projections to derive the asymptotic
distribution of a sequence of variables by comparing these to projections
of a simple form. Conditional expectations are special projections. The
Hajek projection is a sum of independent variables; it is the leading term
in the Hoeffding decomposition.
11.1 Projections
A common method to derive the limit distribution of a sequence of statistics Tn is to show that
it is asymptotically equivalent to a sequence Sn of which the limit behavior is known. The
basis of this method is Slutsky's lemrna, which shows that the sequence Tn == Tn - Sn + Sn
converges in distribution to S if both Tn - Sn O and Sn S.
How do we find a suitable sequence Sn ? First, the variables Sn must be of a simple fonn,
because the limit properties of the sequence Sn must be known. Second, Sn must be close
enough. One solution is to search for the closest Sn of a certain predetermined form. In
this chapter, "closest" is taken as closest in square expectation.
Let T and {S : S E S} be random variables (defined on the same probability space)
with finite second-moments. Arandom variable S is called a projection of T onto S (or
L 2 -projection) if S E S and minimizes
S r-+ E (T - S) 2 ,
S E S.
Often S is a linear space in the sense that a I S I + a2 S2 is in S for every al, a2 ER, whenever
Sl, S2 E S. In this case S is the projection of T if and only if T - S is orthogonal to S for
the inner product (Sl, S2) == ES I S2. This is the content of the following theorem.
11.1 Theorem. Let S be a linear space of random variables with .finite second moments.
Then S is the projection of T onto S if and only if S E S and
E(T - S)S == O,
every S E S.
Every two projections of T onto S are almost surely equal. If the linear space S contains
the constant variables, then ET == ES and cov(T - S, S) == O for every S E S.
153
154
Projecr;ons
Figure) I.J. A variable T and irs projcc(ion S on :tlincar p::Jce.
Prooj. for any S and S in $,
E(T - sfJ. = E(T - 5)2 + 2E(T - S)(S - S) + E(S - S>,2.
1f S satisfies the orthogonality condition, then thc middJe tenn is lero. and we conclude
that E(T - 5)2 ;::. E(T - .5)2, with suict incquaJity unless E(S - 5)2 = O. Thus. the
orthogonaJity condition implics that S is a projection. and 3JsO thnt it is uniquc.
Con\'ersely. for any numbcr ex.
B(T -s - aS)2 - E(T - .5)2 = -2aE(T - S)S + ex 2 ES 2 .
If S is a projection, then this ćxpression is nonnegali\'c for e\'ery cx. But the pambora
ex t- ex 2 ES2 - 2exE(T - S)S is nonnegative ir and only ir the orthogonality condition
E(T - S)S = O is satisfied.
Ifthe constants are in S, then the orthogona1ity condition implies E(T -S)c = O. whcnce
the Jast assertions of the thcorem follow. .
The theorem does not assert that projections always exist. This is not true: The in-
fimum infs E(T - 5)2 need not be achievcd. A sufficient condition for existence is that
$ is closed for the sccond-moment nonn. but existcnce is usuaJly more casily cstablishcd
directly.
,
The orthogonality of T - S and S yierds the Pythagorean rore ET = E(T - S):! + ES-.
(Sce Figure 11.1.) Ir the constants arc containcd in $, then this is also true for \'arianccs
instead of sccond momcnts.
Now suppose a .sequcncc of tatistics Tn and linear spaces SPI is given. For each n. let S"
he the projection of T" on S". Then the limiting bcha\ior of the scquencc T" follows from
that of S'J' and vice \'crsa. providcd the quoticnt varT"!varS,, convcrges to I.
11.2 TJzeorcm. ul S" be linellr J1Jllces of random \-'adab/es U'ithfinite seco"d 11l0mellts
that co1ltaill t/Ze C01l.flmlls. Let T" be raJldom \'adlr/JIes with projecrions S" onto S". Jf
varT"/varS,, -.. I tireli
T" - ET" _ n - ,. O.
sd T" sd S"
J J.2 CVlldiliollol E.xpeClaI;OIl
155
Prao/. \Ve shali prove convergence in second mcan. which is strongcr. The cxpcctation
of the diffcrcncc is lero. hs variance is cquaJ lo
2 _ 2 COv (T n . oS,,) .
sd T" sd "
By the orthogonaJity of Tn - S,. and SnI ii follows that E7;,Sn = ES;. Becausc the constants
are in s,ro this implics that co\'(7;,. S,,) = varS n . and the thcorem folloW5. .
The condition v 'rT"/varS,, -. 1 in the thcorcm implies that the projcctions S" are
asymptotically of the same size as the original Tn. This explains that "nothing is Jost"' in
the limit. and that the differcncc bctwccn T" and its projection convcrges to lero. In the
prcccding thcorcm it is csscntial that the S" arc the projections of the variables Tn. bccause
Ihccondiljon varTn/varSn I for generni sequences S" and T" does not imply anything.
11.2 Conditional Expectation
1l1e expectation EX of arandom varjable X minimizes the quadr'Jtic form (I H- E(X - a)1
over the real numbers a. This may be expressed as follows: EX is the best prediction of X.
given a quadratic loss function. and in the absence of additio'nal infomlation.
The COIllJitiOlWI ,'(I'cctation E(X I Y) of a mndom variable X given arandom \'cctor Y
is dcfined as the best "prediction" of X given knowledge of Y. fonnal1y. E(X I Y} is a
mcasurable function goe Y) of Y that minimizcs
E(X - g(y»)2
over all mcasurable functions g. In the tcm\inology of the preceding scction, E(X I Y) is
the projection of X onto the linear space of all mcasurable functions of Y. h follows that
the conditional expectation is the unique measurable function E(X I Y) of Y that satisfics
the orthogonaJity rclation
E(X - E(X I Y))g(Y) = O.
c\'ery g.
IfE(X I Y) = go(Y}, thenitiscustomarytowriteE(X I Y =)') forgo(y). This js interprctcd
as the expected value of X given that Y = y is obscr\'cd. By Theorcm 11.1 the projcclion
is unique only up to changes on sel., of probabiJity zcro. This mcans that the function Ko()')
is uniquc up to scts B of v:!lues'y such that P(Y e B) = O. (Thcsc could be vcry hig set.,.)
The folJowingexamplcs gi\'c SOn1e propcrtics und also dcscribe the rclationship with
conditional densities.
11.3 Example. The orthogonality relationship with g E I yields the formula EX =
EE(X I Y). Thus. ..the cxpcctaLion of a conditional cxpcctation is the expcctation:' O
11.4 Exal1lple. If X = f(Y) for a measumblc function f. then E(X I Y) = X. This
follows immcdiately from the dclinition, in which the minimum can bc reduced to rero.
The interpretation is that X is perfectly prtdictable gi\'cn know1cdge of Y. O
156
ProJections
11.5 Example. Suppose that (X, Y) has a joint probability density f (x, y) with respect
to a (5 - finite product measure JL X 1), and let f (x I y) == f (x, y) / fy (y) be the conditional
density of X given Y == y. Then
E(X I Y) = 1 xf(x I Y) dfL(X).
(This is well defined only if fy (Y) > O.) Thus the conditional expectation as defined above
concurs with our intuition.
The formula can be established by writing
E(X - g(Y))2 = 1 [I (x - g(y))2 f(x I y) dfL(X)] Jy(y) dv(y).
To minimize this expression over g, it suffices to minimize the inner integral (between sq-
uare brackets) by choosing the value of g(y) for every y separately. For each y, the integ-
ral j(x - a)2 f(x I y) dJL(x) is minimized for a equal to the mean of the density x r-+
f(xly). D
11.6 Example. If X and Y are independent, then E(X I Y) == EX. Thus, the extra knowl-
edge of an unrelated variable Y does not change the expectation of X.
The relationship follows from the fact that independent random variables are uncorre-
lated: Because E(X - EX)g(Y) == O for all g, the orthogonality relationship holds for
go(Y) == EX. D
11.7 Example. If f is measurable, then E(f(Y)X I Y) == f(Y)E(X I Y) for any X and
Y. The interpretation is that, given Y, the factor f (Y) behaves like a constant and can be
"taken out" of the conditional expectation.
Formally, the rule can be estab1ished by checking the orthogonality relationship. For
every measurable function g,
E(f (Y)X - f (Y)E(X I Y)) g(Y) == E( X - E(X I Y)) f (Y)g(Y) == O,
because X - E(X I Y) is orthogonal to all measurable functions of Y, including those of the
form f(Y)g(Y). Because f(Y)E(X I Y) is a measurable function of Y, it must be equal to
E(f(Y)X I Y). D
11.8 Example. If X and Y are independent, then E(f(X, Y) I Y == y) == Ef(X, y) for
every measurable f. This rule may be remembered as follows: The known value y is
substituted for Y; next, because Y carries no information concerning X, the unconditional
expectation is taken with respect to X.
The rule follows from the equality
E(J(X, Y) - g(Y))2 = 1 1 (J(x, y) - g(y))2 dPx(x) dPy(y).
Once again, this is minimized over g by choosing for each y separately the value g (y) to
minimize the inner integral. D
11.9 Example. For any random vectors X, Y and Z,
E(E(X I Y, Z) I Y) == E(X I Y).
11.4 Hoeffding Decomposition
157
This expresses that a projection can be carried out in steps: The projection onto a smaller
set can be obtained by projecting the projection onto a bigger set a second time.
Formally, the relationship can be proved by verifying the orthogonality relationship
E(E(X I Y, Z) - E(X I Y) )g(Y) == O for all measurable functions g. By Example 11.7, the
left side of this equation is equivalent to EE(Xg(Y) I Y, Z) - EE(g(Y)X I Y) == O, which
is true because conditional expectations retain expectations. D
11.3 Projection onto Sums
Let XI, . . . , X n be independent random vectors, and let S be the set of all variables of the
form
n
Lgž(X ž ),
ž=l
for arbitrary measurable functions gž with Eg;(X ž ) < 00. This class is ofinterest, because
the convergence in distribution of the sums can be derived from the central limit theorem.
The projection of a variable onto this class is known as its Hajek projection.
11.10 Lemma. Let XI, . . . , X n be independent random vectors. Then the projection of
an arbitrary random variable T with finite second moment onto the class S is given by
n
S == LE(T I Xž) - (n - l)ET.
ž=l
Proof. The random variable on the right side is certainly an element of S. Therefore, the
assertion can be verified by checking the orthogonality relation. Because the variables X ž
are independent, the conditional expectation E(E(T I X ž ) I X j) is equal to the expectation
EE(T I X ž ) == ET for every i i= j. Consequently, E(S I X j) == E(T I X j) for every j,
whence
E(T - S)gj(X j ) == EE(T - S I Xj)gj(X j ) == EOgj(X j ) == O.
This shows that T - S is orthogonal to S. .
Consider the special case that XI, . . . , X n are not only independent but also identically
distributed, and that T == T (X 1, . . . , Xn) is a permutation-symmetric, measurable function
of the X ž . Then
E(T I X ž == x) == ET(x, X 2 , ..., Xn).
Because this does not depend on i, the projection S is also the projection of T onto the
smaller set ofvariables of the form I:7=lg(X ž ), where g is an arbitrary measurable function.
*11.4 Hoeffding Decomposition
The Hajek projection gives a best approximation by a sum of functions of one X ž at atime.
The approximation can be improved by using sums of functions of two, or more, variables.
This leads to the Hoeffding decomposition.
158
ProjcctiollS
Because a projcction onto a sum of orthogonal spaccs is the sum of the projections
onto the individual spaces. it is convenient to dL ompose the proposed projection space
into a sum of onhogonal spaccs. Given independent variabJcs X.. .... X" and a subset
A C {I t . . . t ,,}. Jet 11" . denote the set of all squarc.intcgrabJc mndom variables of the
ty
8,,(X, :; € A).
for measurable functions 8" of lA I argumente; such that
E(gA(Xl:i E A) I XJ:j E B) = O.
evcry B: IBI < IAI.
(Define E(T J ) = ET.) By the independence of XI. .... XII the condilion in the last
dispJay is automatically valid for any B C {l. 2. . . . . ,,} that docs not contain A. Con.
scqucntly. the spaccs HA. whcn A ranges over all subset of {I. . . . . n}. are pairwise
orthogonal. Stated in its present fonn. the condition rcflects the intcntion to buifd ap-
proximations of increasing complexity by projecting a gi\'cn variabJe in tum' onto the
spaces
[I].
[ glil(XI)J
[ LEgI/.it(X,. Xj) ] .
'<J
where 81;J € H'iJ. g\i.jJ E J1(/.J,..and so fonh. Each new space is choscn orthogonar to the
preccding spaces.
Let PAT denote the projcction of T onto H". Then. by the orthogonality of the II".
the projection onto the sum of the first r space s is' the sum LJ"J ' PAT of the projcctions
on to the individual spaccs. The projection onto the sum of the first two spaccs is the Hajek
projection. !vIore gencrally. the projections of zero. first. and sccond order can be seen
to be
PttT = ET.
PIO T = E(T I Xi) - ET,
P1'.JIT = E(T I Xi, X j )- E(T I Xi} - E(T IX}) + ET.
Now the general fonnula given by the following lemma should nol he surprising.
11.11 Le11lrna. ut XI. . .., X n be independent ralIdom \'ariables, and let T be tlll arbi-
trar)' random "ariable u'ith ET 2 < 00. Then the projeclion ofT onlO H" i.f gi,'ell by
p"T= L(_I)'AJ-J8iE(T IXi:i ED).
HCA
Jf T J. Ila for e\'er}' sllb et B C A of a gi,"e"sel A. then E(T I XI : i E A) = O. COlIse.
qltenlly. the sIIm of the .'ipaces Ha w;III B C A C01lralll.f all.fiqllare-imegrllblejllllctiolls of
(Xi: i E A).
Proof. Abbreviatc E(T I Xi : i E A) to E(T I A) and gA (Xi: i E A) to gA. By the indc-
pendence of Xlt' .. X n il follows that E(E(T I A) I B) = E(T I A n B) forevery subsets A
11.4 lIoeffilitr8 Decompo!iiliml
159
and B of (19' . . I Il J. Thus 9 for Pit Tas defined in the lemma and a set C strictly contained
in A,
E(PAT I C) = L(_1)IAI-IHIE(T 18 n c)
OCA
'''1-\("1 ( I A I I C I)
= L L (-I ),ltl-,Đl- j E(T I D).
OcC J=U J
By the binomial fonnula, the inner sum is lCro for e\'cry D. Thus the left side is lero. In
vicw of the fonn of PA T. it was not a loss of generaJity to assume that C c A. Hence P" T
is containcd in ilA.
Ncxt we \'crify the orthogonality rclationship. For any measurable function 8,\,
E(T - PAT)g" = E(T - E(T I A»g" - L (_I)IAI-IBIEE(T I B)E(gA I B).
JlC
IJf
This is zcro for any gA E HA' This concludes the proof that PA T is as gi\'cn.
\Vc provc the sccond asscrtion of the lemma by induction on r = lA,. If T .1 H I
the n E(T I ) = ET = O. Thus the assertion is true for r = O. Suppose that it is true
for O, . . _ , r - I, and consider a set A of r elements. ff T ..L Hil for every B C At then
ccrtainly T 1. He for evcry C C 8. Consequcntly, the induction hypothesis show s that
E(T I Il) = O for c\'cry B C A of r - I or fewer elements. The fonnula for PAT now shows
that p,\ T = E(T 1 A). By assumption the left side is lero. This conc1udcs the induction
argument.
The final assertion of the temma foUows ir the variable T. := T - L8CA PR T is zcro
for evcry T that dcpends on (X, : i E t\) only. But in this casc TA depcnds on (X, :; e A)
only and hence equals Err,\ t 1\), which is uro, becausc TA lo H[l for evcry B C A. .
Jr T = T(X.,..., XII) is pcnnutation-symmelric and Xi..... XII arc independent and
idcnticaJly distributcd. the n the Hocffding decomposition of T can he simplified 10
It
T = L L gr(Xj:i E A).
,. Q 1141="
for
g,(XI,...,X,) = L (_1),.-18 I ET(.t, e B,X; B).
Bel 1.....'1
The inncr sum in the reprcsentation of T is for each raU -statistic of order r (as discussed
in me Chapter 12), with degenerate kernel. All terms in the sum are orthogona1. whcncc
the variance of T can be found as varT = E;.I (;)Eg;(X....., X,).
Notcs
Orthogonal projcctions in Hilbcrt spaces (completc inner product spaces) arc a classica1 sub-
jeca in functional analysis. We have limited ourdiscussion to the liilbert space L2(Q, U, P)
of all square-integrable random \'ariables on a probability space. Another popular method to
160
Projections
introduce conditional expectation is based on the Radon-Nikodym theorem. Then E(X I Y)
is naturally defined for every integrable X. Hajek stated his projection lemma in [68] when
proving the asymptotic normality of rank statistics under altematives. Hoeffding [75] had
already used it implicitly when proving the asymptotic normality of U -statistics. The "Ho-
effding" decomposition appears to have received its name (for instance in [151]) in honor
of Hoeffding's 1948 paper, but we have not been able to find it there. lt is not always
easy to compute a projection or its variance, and, if applied to a sequence of statisties, a
projection may take the form L gn (Xi) for a function gn depending on neven though a
simpler approximation of the form L g(X i ) with g fixed is possible.
PROBLEMS
1. Show that "projecting decreases second moment": If S is the projection of T onto a linear space,
then ES 2 < ET 2 . If S contains the constants, then also var S ::s varT.
2. Another idea of projection is based on minimizing variance instead of second moment. Show
that var(T - S) is minimized over a line ar space S by S if and only if cov(T - S, S) == O for
every S E S.
3. If X > Y almost surely, then E(X I Z) > E(Y I Z).
4. For an arbitrary random variable X > O (not necessarily square-integrable), define a conditional
expectation E(X I Y) by limM oo E(X /\ M I Y).
(i) Show that this is well defined (the limit exists almost surely).
(ii) Show that this coincides with the earlier definition if EX 2 < 00.
(iii) If EX < 00 show that E( X - E(X I Y)) g (Y) == O for every bounded, measurable function g.
(iv) Show that E(X I Y) is the almost surely unique measurable function of Y that satisfies the
orthogona1ity relationship of (iii).
How would you de fine E(X I Y) for arandom variable with EIX\ < oo?
5. Show that a projection S of a variable T onto a convex set S is almost surely unique.
6. Find the conditional expectation E(X I Y) if (X, Y) possesses a bivariate normal distribution.
7. Find the conditional expectation E(XI I X(n)) if XI, ..., X n are arandom sample of standard
uniform variables.
8. Find the conditional expectation E(X 1 I X n ) if XI, . . . , X n are i.i.d.
9. Show that for any random variables S and T (i) sd(S + T) < sd S + sd T, and (ii) I sd S - sd T I ::s
sd(S - T).
10. If Sn and 1',1 are arbitrary sequences of random variables such that var(Sn - Tn) jvarT n -+ O,
then
Sn - ES n
sdS n
Tn - ETn p
-+ O.
sdTn
Moreover, var Sn jvarT n -+ 1. Show this.
11. Show that PAh(Xj : Xj E B) == O for every set B that does not contain A.
12
U-Statistics
One-sample U -statistics can be regarded as generalizations of means.
They are sums of dependent variables, but we show them to be asymptoti-
cally normal by the projection method. Certain interesting test statistics,
such as the Wilcoxon statistics and Kendall' s i -statistic, are one-sample
U -statistics. The Wilcoxon statistic for testing a difference in location
between two samples is an example of a two-sample U-statistic. The
Cramer-von Mises statistic is an example of adegenerate U-statistic.
12.1 One-Sample U-Statistics
Let X }, . . . , X n be arandom sample from an unknown distribution. Given a known function
h, consider estimation of the "parameter"
e == Eh (X }, . . . , X r ) .
In order to simplify the formulas, it is assumed throughout this section that the function
h is permutation symmetric in its r arguments. (A given h could always be replaced by
a symmetric one.) The statistic h(X}, . . . , X r ) is an unbiased estimator for e, but it is
unnatural, as it us es only the first r observations. A U-statistic with kernel h remedies this;
it is defined as
1
U = (;) h(X/31"'" X/3),
where the sum is taken over the set of all unordered subsets tJ of r different integers chosen
from {I, . . . , n}. Because the observations are i.i.d., U is an unbiased estimator for e
also. More over, U is permutation symmetric in X}, . . . Xn, and has smaller variance than
h(X},..., Xr). In fact, if X(1),..., X(n) denote the values XI,..., X n stripped from their
order (the order statistics in the case of real-valued variables), then
U == E(h(X I , .. ., Xr) I X(1), ..., X(n»).
Because a conditional expectation is a projection, and projecting decreases second mo-
ments, the variance of the U -statistic U is smaller than the variance of the naive estimator
h(X},..., Xr).
161
162
U-Stat;sti<"s
In this scction il is show n that the scquence .fii(U - 6) is asymptoticaJly nonnal under
the condirion that Eh 2 (X I, . . . , X,) < 00.
12.1 Example. A U-statistic of dcgrce r = I is a mean ,,-I Er:..I"(X;). The assel1cd
a5)'mptotic nonnality is the n just the central limit theorem. O
12.2 Example. For the kemelll(:c., x:!) = 4<xl - xZ)2 of degree 2. the pal.1metcr () =
EIz(X I. X2) = var X 1 is the variancc of the obscrvations. The corrcsponding U -statistic
can bc calculated to be
1 ",,,,1 ., I -.,
U = n L.J L...J-(Xi - Xj)- = - L...J(X, - X)",
(2) I<.j 2 n-I ;=1
Thus. the sample variance is a U -statistic of order 2. O
The asymptotic normality of a sequence of U-statistics. ir" -+- 00 with the kcmcl
rcmaining fixcd, can be estabJished by the projection mcthod. The projcction of U - () onto
the set of all statistics of the fonn L:?...I.':; (Xi) is gi\'cn by'
n r N
il = LE(U - () I Xi) = - Lhl(X,),
1=1 11 I=J
wherc the function ", is given by
h,(x) = Eh(x, X2,..., X,) - O.
The first equaJity in lhe formula for U is the H jek projec1ion principle. The second cquality
is establishcd in the proof bclow.
The scqucncc of projcetions cl is as)'rnptoticalJy nom1al by the central limit theorein,
provided Ehi(X I) < 00. The difference bctwccn U - O and its projcction isasymptotically
ncgligible.
12.3 Theorem. 1/ EIl 2 (X,. .. ., X r ) <: 00, lire" ..[ii(U -. O - il) O. COflseqllem(\',
the ,'ieqlle1Zce ,Jii(U - O) is as)'mplotical/)' nOTmal with nrecm O and wzr;ance r 2 (1' w!tere,
.....ith X lo . . . , X n X J ' . . . , X; de1Zoting U.cl. \'ariables,
(I = cov(h(X.. X2,..., X,). Jr(X.. X;..,.. X;»).
Proof. \Ve first vcrify the fonnula for the projcction iJo Il sufficcs 10 show that E(U -
() I XI) = Iz, (X,). By the independenee of the obsef\'ations and pcnnutation symmetry of
Jr,
( . ) { Izl (.t) if; e p
E Ir(XtJ...... XI,,) - 81 Xi = X = O if i <t fl.
To calculate E(U - 81 Xi), we take the averagc over all {J. Then the first case occurs for
( ::> of the vectors {3 in the definition of U. The factorr/n in the fonnuJa for the projcction
(; arises as r /" = ( : : )/( ).
/2./ OTlt'-Sanrplt' U-Stal;Slic's
163
The projcction il has me an zcro. and \'anance equal to
.,
" r" ') ,
varU = -Ehj(X.)
"
r 2 f r2
= - E(h(x. X 2 . .... X,) - 9)Eh(x. X;..... X;>dPx,(x) = - I'
n II
Because this is finitc. the sequence ./ii D convergcs weakly to the N (O. r 2 I )-distribulion
by the central limit Lhcorcm. By Thcorem I 1.2 and Slutsky's Icmma. the sequencc (U -
O - U) convcrgcs in probability to lero. provided var U Ivar (j -+ I.
In view orthe permutation symmetry of the kemel h. an express ion ofthe type cov(Iz (X Ifl'
. . . . X /f.). Iz (X If;' . . . t X If;»). depcnds. only on the numbcr of variablcs X. that arc common
to Xpl....' X p , and XJ1j...., Xp;. Let (' be this covariancc if c variables are in common.
Then
var U = ( ; ) -2 I: I: cov{Iz(X".. ... t XfJ.). h(XlJj. .... XI'; »
= (;r (;) ( ) (;= ) <.
The last stcp follows. bccause a paie (fJ. tJ') with c indexes in common can be chosen by first
choosing the r indexes in fi. next the c common indcxes from /3. and finally the remaining
r - c indcxcs in tJ' from {I. . . . . II} - /1. The expression can be simplified to
I: ' r!2 (n - rHIl - r - I)... (n - 2r + c + 1)
varU = , (("
/";;:1 c!(r-c)!" n(n -1)...(n -r+ I)
In this sum the first term is O(I/n). the second term is O(I/"z), and so forth. Be-
cause Il times the tirst term converges to r2 1. the desired limit rcsull \'ar U/\'arU
follows. .
12.4 Example (Sigllcd rOllk s/a/is/ic). The parameter O = P(X. + X2 > O) corresponds
to the kemel Iz(XI.X2) = I{x, +X2 > O}. The corresponding U-statistic is
I
U = ( II ) L L I{X, + Xj > O).
1 .</
1l1is statistic is the average numbcr of pairs (X,. X J) with positive 5um X; + X j > O. and
can he used as a test statjstic for invcstig.ltjng whcthcr the distribution of the obscrvations
is located at zero. If many pajrs (X,. XJ) yieJd a positivc sum (rclative to the total numbcr
of pairs), then wc have an indication that the distrjbution is centcred to the righl of zero.
The scqucncc .jii(U - O) is asymptotica1Jy noroJal with mcan zcro and \'3riance 4('1. If
F denotes the cumulativc djstribution function of the obscrvations. then the projection of
U - (J can he wriUen
2 fI
(j = -;; I:(F(-X;) - EF(-X i »).
.-i
This formula is useful in subscquent discussion and is also convenient to express the asymp-
totic variance in F.
164
U-Statistics
The statistic is particularly useful for testing the nulI hypothesis that the underlying
distribution function is continuous and syrnmetric about zero: F (x) == 1 - F ( - x) for
every x. Under this hypothesis the parameter e equals e == 1/2, and the asymptotic variance
reducesto4var F(X 1 ) == 1/3, because F(X 1 ) isuniformlydistributed. Thus, underthenulI
hypothesis of continuity and symmetry, the limit distribution of the sequence -Jfi(U -1/2) is
normal N(O, 1/3), independent of the underlying distribution. The last property means that
the sequence U n is asymptotically distribution free under the nulI hypothesis of symmetry
and makes it easy to set critical values. The test that rejects Ha if ,J3n(U - 1/2) > Za is
asymptoticalIy of level a for every F in the null hypothesis.
This test is asymptoticalIy equivalent to the signed rank test of Wilcoxon. Let Ri, . . . ,
R; denote the ranks of the absolute values IX 11, . . . , IX n I of the observations: Rt == k
means that I Xi I is the kth smalIest in the sample of absolute values. More precisely, Rt ==
L =11 {I X ji < I Xi I}. Suppose that there are no pairs of tied observations Xi == X j. Then
the signed rank statistic is detined as W+ == L7=1 Rt1{Xi > O}. Some algebra shows that
w+ = (;) U + t1{X; > O}.
The second term on the right is of much lower order than the first and hence it follows that
n- 3 / 2 (W+ - EW+) 'v'7 N(O, 1/12). O
12.5 Example (Kendall's L). The U -statistic theorem requires that the observations X 1,
. . . , X n are independent, but they need not be real-valued. In this example the observations
are a sample of bivariate vectors, for convenience (somewhat abusing notation) written as
(X 1, YI), . . . , (X n , Y n ). Kendall's i -statistic is
4
i = n(n -1) LI: 1{(Y j - Y;)(X j - Xi) > O}-1.
l<}
This statistic is a measure of dependence between X and Y and counts the number of
concordant pairs (Xi, Yi) and (X j, Y j ) in the observations. Two pairs are concordant if the
indicator in the detinition of i is equal to 1. Large values indicate positive dependence (or
concordance), whereas smalI values indicate negative dependence. Under independence of
X and Y and continuity of their distributions, the distribution of i is centered about zero,
and in the extreme cases that all or no ne of the pairs are concordant i is identically 1 or
-1, respectively.
The statistic i + 1 is a U -statistic of order 2 for the kernel
h ( G ) , G ) ) = 2I{ (Y2 - YI)(X2 - XI) > O}.
Hence the sequence -Jfi( i + 1 - 2P( (Y 2 - Y 1 )(X 2 - XI) > O)) is asymptotically normal
with mean zero and variance 4?;1. With the notation Fl(x, y) == P(X < x, Y < y) and
Fr(x, y) == P(X > x, Y > y), the projection of U - e takes the form
U = 4 t(F l (X i , Yi) + rex;, Yi) - EFl(X;, Yi) - Er(X i , Yi))'
n. 1
l=
If X and Y are independent and have continuous marginal distribution functions, then
E i == O and the asymptotic variance 4?; 1 can be calculated to be 4/9, independent of the
12.2 Two-Sample U-statistics
165
y
discordant
()
concordant
x
Figure 12.1. Concordant and discordant pairs of points.
marginal distributions. T hen -Jfi T -v--+ N (0,4/9) which leads to the test for "independence":
Rejectindependenceif ,.j 9nj4lTI > Za/2' D
12.2 Two-Sample U-statistics
Suppase the observations consist of two independent samples XI, . . . , X m and YI, . . . , Y n ,
i.i.d. within each sample, from possibly different distributions. Let h (XI, . . . , X r , YI, . . . , Ys)
be a known function that is permutation symmetric in XI, . . . , X r and YI, . . . , Ys separately.
A two-sample U -statistic with kemel h has the form
1
U = ( )( ) h (X",] , . . . , X"'r' YtJ] , . . . , YtJJ,
where ex and f3 range over the collections of all subsets of r different elements from
{1, 2, . . . , m} and of s different elements from {1, 2, . . . , n}, respectively. Clearly, U is
an unbiased estimator of the parameter
e == Eh (X 1, . . . , X r , YI, . . . , Y S ).
The sequence Um,n can be shown to be asymptotically normal by the same arguments as
for one-sample U -statisties. Here we let both m --+ 00 and n --+ 00, in such away that the
number of Xi and Y j are of the same order. Specifically, if N == m + n is the total number
of observations we assume that, as m, n --+ 00,
m
---+A
N '
n
---+l-A
N '
O < A < 1.
To give an exact meaning to m, n --+ 00, we may think of m == mv and n == nv indexed by
a third index 1) E N. Next, we let mv --+ 00 and nv --+ 00 as 1) --+ 00 in such away that
mv/ N v --+ A.
The proj ection of U - e anto the set of all functions of the form :Lr= 1 k i (Xi) + :L = Il j (Y j )
is gi ven by
A r s
U == - h1,O(Xi) + - hO,l(Yj),
m. I n. 1
l= J=
166
U.Star;slics
wherc the functions "1.0 and ho.1 are defincd by
"I.O(.\') = Eh (x t X 2. . . . t X,. YIt . . . . Y,,) - O.
1z0.1 ()') = Eh(Xr..... X n y. Y2. .... Y,) - O.
This follo\\'s. as before. by first applying the H5jek projection lemma. and next exprcssing
E(U I Xi) and E(U I Yj) in the kemel function.
If the kemcJ is square-integrablc, thcn the sequence cl is asymptoticalJy nonnal by the
centml limit theorem. The diffcrencc between (j and U - (J is asymptoticalJy ncgligiblc.
12.6 Tlzcorem. JfEh 2 (X J t .... X r , Y J t .. .. YI) < 00. thclllhescqllcllce JN(U -O-U)
con\"erge.r in probabi/it}" to zero. 'Collseqllcllll) the seqllcm:e .J'N(U - O) ("01n'crges in
di.\"lributioll 10 the llonlltl!!aw with metl/l zero tl1llIl'Or;ol1ce r2 Lo/). + s2('o.d( I - ).).
,,,,'Izere. with tlze Xi being U.d. \'ariable.'i indepelldent of the U.d. \.'{lriablcj' YI'
(.J = cov(h(X..... t X" yI...., Y..).
h(X...... X(', X;+I'.... X;. rit.... r J . Y +I"'" r;»).
Proo/. The argument is simiJar to the onc gi\'cn prcviously for onc-sample U -statistics.
The variances of U and its projection arc given by
..,2 s2
varU = -("l.O + -('0.1
m Il
U l ' ( m ) ( r ) ( m - ' ) ( Il ) ( s ) ( " - S )
var = ( ,)2(:)2 r c r - c: .f d. s _ ti ('.J.
It can he checked from this that both the sequcnce N\'3r D and the scqucncc Nvar U
con\'erge to the number r2 I.O/A + S2 O.I/( I - J.). .
12.7 Example (t.faml- U1,itllc)' statistic). The kcmcl for the paramcter O = P(X === Y) is
"(x, y) = 1 IX YI. which is of ordcr I in both x and)l. The corrcsponding U-statiMic is
I "'''
U = - l:LJ{X':5 Yi}'
Inll ,=. 1=1
The statistic nlll U is known as the J,lam,- "''';I11('Y statistic and is uscd to test for a difrerence
in location between the two samples. A Jargcvalue indicatcs that the Y J arc "stochastically
largcr" than the X,.
Ir the X, and Y J have cumulative distribution functions F and G. rcspcctiveJy. the n the
projcction of U - O can bc wriuen
1 m I
U = -- L(G-(X,) - EG_(X;») + - L(F(Y/) - EF(Y;»).
m i=1 Il j':2l
It is casy to obtain the limit distribution orthe projections D (and hence of U) from this for-
mula. In particuJar. under the nuli hypothcsis that the pooJed sample XI. . . . . X m1 YI. - . .. Y"
is i.i.d. with continuous distribution function F = G. the sequcnce J2111 1l IN (U - J/2)
12.3 D('gmff(lr U.Swri.\'Iic.f
167
con\'erges to a standard nonnal djstributjon. (The pammeter equals (} = 1/2 and ("0.1 =
("1.0 = 1/12.>,
Ir no observations in the pooled sample arc tjed, then mIJU + 4"(n + I) is cqual to the
sum of the ranks of the Yj in the pooled s3mple (see Chapter 13). Hence the latter slHtistic.
the WUCOX()/J lu.'o-.mmplc SICllislie. is asymptotically nom1al as well. O
*12.3 Degencrate U-Statistics.'
A sequence of U -statistjcs (or, bctter, their kemel function) is called dege-nerare if the
asymptotic \'ariance r 2 {1 (found in Theorem 12.3) is zero. The fonnula for the variance
of a U -statjstic (in the proof of Thcorcm 12.3) shows that var U is of the ordcr n-I' if
{I = .. . = I'-I = O < ("c' In this casc, the sequencc ".'IZ(U n - O) is asymptotically tight.
In this section we derive its limit distribution.
Considcr the Hoeffding decomposition a.s discusscd in scction 11.4. For a U -statistic
Un with kemellz of order r, based on obscrvatjons XI. . . . . X'I' thi can be simplified to
, 1 ,. ( r )
U" = L: L ( II ) L PAh(XfJJ...., XIfr) = L (' Uri."
c=0 IAI=(' r JI t-==O
(sa}').
nere, for each O :s c :s r. the variable U".t' is a U -statistk of order c with kernel
,,("(X I, ... . X() = PII.....dlz(X J, . ... X,).
To scc this fix a set A with c elements. Because the space HA is orthogonal toa11 functions
g(Xj:j e B) (i.e., the space LecB Ile) for cvcry ct B that docs not contain A, the
projcction PAIz(Xp,.... ,Xp.) is zero unless A C tJ = IP".... Pr}' For the remaining
p the projcction PA Iz (XPI ,.... X,.,) doe s not depcnd on jJ (i.e.. on the r - c clements of
jJ - A) and is a fixcd function hto of (X j : j e A). This follo\\'s by symmetry. or cxplicitly
from the fommla for the projections in section 11.4. The function ,, is indeed the function
as given prcviously. There are ( : ) vectors tJ that contain the set A. The claim that U".r: is
a U -statistic with kemel hr: now follows by simple algebra. using the fact that ( : :]/( ,)(:)
= 1/( ).
By the defining properties orthe space 1/(I. t'l' il fol1ows that the kemellz is degenerate
for c 2: 2. (n facto it is s/rollg/)' Jegeller£lle in the sense that the conditional cxpcctation of
hc(X I. . . . . Xc) gi\'en ,my stricl subset of the variabies XI. . . . , X(' is zcro. In other words,
the integrnl J Iz(x. X2. ..., Xc) d P(:c) with respect to any single argument vanishes. By
the same reasoning. U".t' is uncorreJated with e\'ery measurable function that dcpcnds on
strictly fewer than (" elements of XI, . . . . X n'
Vole shali show that the sequence 11('/2 Un.C' converges in distribution to a limit with variance
c! FJI;(X I. . . . . X(") for every c I. Then it follows that the sequence n('/2(U,,' - O)
converges in distribution for c equal to the smallesl value such that he 1= O. For c 2 the
limjt distribution is not nomta! but is known as Gauss;tl/l clzaos.
Bccausc the idea is simple, but the statement of the theorem (apparently) neccssarily
complicatcd, first consider aspccial casc: c = 3 and a "product kemel" of the' fonn
IrJ(XI. .t;!. X3) = II (xl)/2(x2)b(X3)'
168
U-Statistics
A U -statistic corresponding to a product kernel can be rewritten as a polynomial in sums
of the observations. For ease of notation, let JPnl == n-I 'L7=I/(X i ) (the empirical mea-
sure), and let Gnl == ,Jn(JlD n - P)I (the empirical process), for P the distribution of the
observations XI, . . . , Xn. If the kemel h3 is strongly degenerate, then each function li has
mean zero and hence Gnl i == ,JnJPnl i for every i. Then, with (il, i 2 , i 3 ) ranging over all
triplets of three different integers from {I, . . . , n} (taking position into account),
3 ! ( n ) 1
n 3 / 2 3 U n ,3 = n 3 / 2 LU) hi)) iI (Xi)) h (XiJ h (Xi))
== GnlIGnl2Gnl3 - JlD n (/I/2)G n I 3
rp n (fI f2/3)
- W'nUjh)Gnh - W'n(hh)GniI + 2 ,Jn .
By the law of large numbers, JP n p lalmost surely for every I, while, by the central
limit theorem, the marginal distributions of the stochastic processes I 1---+ G n I converge
wealdy to multivariate Gaussian laws. If {Gf : I E L 2 (X, A, P)} denotes a Gaussian
process with mean zero and covariance function EG/Gg == P 19 - P I Pg (a P-Brownian
bridge process), then G n G. Consequently,
n 3 / 2 U n ,3 Gf I Gf2 G /3 - P(/I/2)Gf3 - P(/I/3)Gf2 - P(f2f3)GfI.
The limit is a polynomial of order 3 in the Gaussian vector (GrfI, G/2, Gf3).
There is no similarly simple formula for the limit of a general sequence of degenerate U -
statistics. However, any kernel can be written as an infinite linear combination of product
kernels. Because a U -statistic is line ar in its kemel, the limit of a general sequence of
degenerate U -statistics is a linear combination of limits of the previous type.
To carry through this program, it is convenient to employ a decomposition of a given
kernel in term s of an orthonormal basis of product kemels. This is always possible. We
assume that L 2 (X, A, P) is separable, so that it has a countable basis.
12.8 Example (General kernel). If 1 == lo, 11, 12, . . . is an orthonormal basis of L 2 (X,
A, P), then the functions IkI X . . . X Ike with (k I , . . . , ke) ranging over the nonnegative
integers form an orthonormal basis of L 2 (X e , AC, p e ). Any square-integrable kernel can
bewrittenintheformhe(XI,..., xe) == 'La(k I ,..., k e )lk I X... X fke fora(k I ,..., ke) ==
(he, fk I X . . . X fke) the inner products of he with the basis functions. O
12.9 Example (Second-order kernel). In the case that c == 2, there is a choice that is spe-
cially adapted to our purposes. Because the kernel h 2 is symmetric and square-integrable
by assumption, the integral operator K : L 2 (X, A, P) 1---+ L 2 (X, A, P) defined by KI (x) ==
J h 2 (x, y) I (y) d P (y) is self-adjoint and Hilbert-Schmidt. Therefore, it has at most count-
ably many eigenvalues Aa, Al, . . . , satisfying 'L A < 00, and there exists an orthonormal
basis of eigenfunctions 10,/1, . . . . (See, for instance, Theorem VI.16 in [124].) The kemel
h 2 can be expressed relatively to this basis as
00
h 2 (x, y) == LAklk(x)lk(Y)'
k=O
For adegenerate kemel h 2 the function 1 is an eigenfunction for the eigenvalue O, and we
can take lo == 1 without loss of generality.
12.3 Degenerate U-Statistics
169
The gain over the decomposition in the general case is that only product functions of the
type f x f are needed. O
The (nonnormalized) Hermite polynomial Hj is a polynomial of degree j with leading
coefficient x j such that J Hi (x)H j (x) ep (x) dx == O whenever i i=- j. The Hermite polyno-
mials of lowest degrees are Ha == 1, HI (x) == X, H 2 (x) == x 2 - 1 and H3 (x) == x 3 - 3x.
12.10 Theorem. Let he : X e 1---+ IR be a permutation-symmetric, measurable funetion
of e arguments such that Eh (X 1, . . . , Xe) < 00 and Eh e (X1, . . . , X e -1, Xe) - O. Let
1 == fa, f1, f2, . . . be an orthonormal basis of L 2 (X, A, P). Then the sequenee of U-
statisties Un,e with kernel he based on n observations from P satisfies
d(k)
n e / 2 U n ,e L (he, fkI x ... x fkJ n Hai(k) (CG1/1i (k)).
k=(ki,...,ke)EN e i=l
Bere G is a P -Brown ian bridge process, the funetions 1/11 (k), . . . , 1/1 d(k) (k) are the different
elements in fkI' . . . , fke' and ai (k) is number of times 1/1i (k) oecurs among fkI' . . . , fke'
The variance of the limit variable is equal to c! Eh (X1, .. ., X e ).
Proof. The function he can be represented in L 2 (X e , AC, p e ) as the series Lk (he, fk i X
. . . , fke) fkI X. . . X fke' By the degeneracy of he the sum can be restricted to k == (k I , . . . , ke)
with every k j > 1. If Un,eh denotes the U -statistic with kernel h(X1, . . . , x e ), then, for a
pair of degenerate kemels h and g,
e! e
cov(U n eh, Un eg) == P hg.
, , n(n - 1) . . . (n - c + 1)
This means that the map h 1---+ n e / 2 .jC! Un,eh is close to being an isometry between L 2 (p e )
and L 2 (p n ). Consequently, the series Lk(he, fki X ... X fke)Un,efk i X ... X fke con-
verges in L 2 (p n ) and equals Un,ehe == Un,e. Furthermore, if it can be shown that the
finite-dimensional distributions of the sequence of processes {Un,e iki X ... X ike: k E
Ne} converge weakly to the corresponding finite-dimensional distributions of the process
{fl: Bai(k) (CG1/1i (k)) : k ENe}, then the partial sums of the series converge, and the proof
can be concluded by approximation arguments.
There exists a polynomial Pn,e of degree c, with random coefficients, such that
c! ( n ) ( )
n e / 2 e Un,e fk i X . . . X ike == Pn,e CG n fk I , . . . , CG n ike .
(See the example for c == 3 and problem 12.13). The only term of degree c in this polynomial
is equal to CG n iki CG n ik 2 . . . CG n ike' The coefficients of the polynomials Pn,e converge in
probability to constants. Conclude that the sequence n e / 2 e! Un,e iki X . . . X fke converges
in distribution to Pe (CG fk i , . . . , CG ike) for a polynomial Pe of degree c with leading term,
and only term of degree c, equal to CG fk i CG fk 2 . . . CG fke' This convergence is simultaneous
in sets of finitely many k.
It suffices to establish the representation of this limit in term s of Hermite polynomials.
This could be achieved directly by algebraic and combinatorial arguments, but then the
occurrence of the Hermite polynomials would remain somewhat mysterious. Altematively,
170
U-Sratb.-r;Cj'
the representation can be derived from the definition of the Hermite polynomials and co-
variance calculations. By thc dcgcncracy of the kernel /J'I X ... x Ji.. the U statistic
U".(.fl.l X ... X It, is onhogonal to all measurable functions of ,. - 1 O( fcwcr elements
of X ,. . . . . X". and thcir 1incar combinations. This includes the functions ni (G n .':, )U, for
arbhrary functions J:j and nonnegativc integcrs ai with L: ti, < C". Taking lirnits. we con-
dude that Pc(GJi.., ..., Gfk..) must bc orthogonal to C\'cry poJ)'nomial in Gfir..... Gfs..
of degree Jcss than c - I. By the orthonormaJity of the basis f,. the variabJes Gf, are
independent standard normal variablcs. Bceausc the Hcrmjrc polynomials fonn a basis for
the polynomials in one variable. thcir (fensor) producl forrn a basis for the poJynomiaJs
of more than onc argument. The poJynomial P.. can he wriucn as ajincar combination of
elements from this basis. By the orthogonality, the cocfficicnrs of base elements of degree
< (' vanish. From the basc elemenl., of degrce c.' only the product as in the theorem can
occur. as foJ1ows from consideration of the lcading tenn of p.,.. .
J2.J J Example. For c = 2 and a basis I = lo. JI. . .. of cigcnfunctions of the kcmcl
"1,. we obtain alimit of the form 'Ldh2. II; X h)lIl.(GI1J. By the orthonormalit)' of the
basis this variable is distribured as 'L At (Zf - I) for ZI. Zlo ... a sequencc ofindepcndcnt
standard nonnaJ variabJes. O
12.12 Examp/e (Sampic 'aria"ce). Thckemel "(.\'1. X2) == (XI-X2)2 yiclds thesample
variancc S,;. Becausc this has asymptotic variance 11 - Jl (sčc ExampJc 3.2), the kernel
is degcncratc ir and only ir JA.t = Jt . This cnn happcn only ir (X I - 0'1)2 is constant. for
0'. = EX,. If we center the observations, so that 0', == O. then this means that X I only
rakes the \'alues -(1 and (1 = .jjI2. each with probabiJity 1/2. This is a very dcgcncrate
situarion. and j( is casy to find the limit distribution directly. but pcrhaps it is instructi\'c to
apply the general theorem. Thekemcls It take the forms (Sec scction 11.4).
ho = E (X, - X.,) = 0'2.
.. ..
IIJ (.\'1) = E (XI - X 2)2 -ul..
"1(.\",.X2) = !<.\'I -X2):! - E (Xt - X2)2 - E!(X. -X1)2 +02.
The ken1cJ is degenerarc if Il, = () aJ most sureJ)'. and then th second-ordcr kernc1 is
1'2(.\',. .\'2) = ('\'1 - X2)2 - u 2 . Bl.'Causc the underlying dhtribution has only two sup-
port points. th eigcnfunctions I of the cOlTCsponding intcgml opcr.1tor can bc idenrified
with \'ectors (/(-0'). I(a») in IR . Some linear algebra shows thott they arc (I. f) md
(-I. I). corresponding to the cigcnvalues O and -(1 . rcspccti\'ery. Corrcsponding.ly. undcr
degcncracy thc kemc1 allows the decomposition
1 ( ' ' ) ' .' ( .\"' )( .\"2 )
"2(Xa. X ;Z) = 2 xi +xi -0- -x,x = -0'-;; ;;.
\Vc Can concJude (hat the scqucncc 11 (S,; - /l.!) con\'crges in dislrihution to _(12(Z - I). O
12.13 F..xample (Cramir-}'on JUises). LctF,,(.t) = 11-' L::'=I J IX, .tl be theempirical
distribution function of a mndom sampic XI. . . . . X" of re;IJ-\'4tlued nmdom \'ariablcs. The
Crtlmer-\ J/l /"fi,\,t!.'i .'italislk for testing the (nuli) hypothcsis that the underlying cumulati....c
Probl m.\'
171
distribution is a givcn function F is gi\'en by
f In " f
" (iF" - F)2 d F = ;;?::L (lx, " - F(x»)(lx,::t - F(.t») d F(x).
,..1 Jcl
The doublc sum restricted to the off.diagonal tcnns' is a U -statistic, with, undcr Ilo. a
degenerate kcme1. Thus. this statistic convcrgcs to a nondcgcncmte limit distribution. The
di&lgonal tenns contribute the constant f F (I - F) d F to the limit tlistribmion, by the law
of large numbcrs. If F is unifonn, then the kcmct of the U -S[ati tic is
h(x. y) = !X2 + !y2 -.{ v)' +.*'
To find the eigenvalues of the corrcspomling intcgral operator K. we diffcrcntiate the identity
Kf = i f twice, to find thecquation)...f"+ f = f [(.f) ds. Because the kerncl isdcgenerate.
the constants arc cigcnfunctions for the cigenvaJue O. The eigenfunctions corresponding to
nonzcro cigenvaluc!\ are orthogonalto this eigcnspace. whcncc f [(s) C/S = O. The cquation
)../ ' + f = O has solutions cos {IX und sin iIX for il:! = )..-1. Rcinscrting thcse in the original
equation or utilizing the relation f !(.r) d.r = O. we find that the oonzero eigenvalues are
cqual to' j-21f-2 for jeN, with cigenfunctions J2 cos Trj.t. 11105. the Cramćr-Von r-.1ises
statistic convcrgcs in distribution to 1/6 + L I j-2;r-2(Z; - I). For anothcr derivation
of the limit distribution. sec Chapter 19. O
Notes
The main part of this chaptcr has its roots in the paper by Hocffding [76]. Bccause the asymp-
totic \'ariance is smaller than the true variance of a U - (atis(ic. Iloeffding recommends to
apply a standard nonnal approximation to (U - EU)I sd U. Degenerate U-statistics were
considcred. among othcrs. in (131) within the contcxt of more gencral lincar combina-
tions of symmctric kcmcls. Arconcs and Gine (21 have studied the we lk convergence of
"U-processes". slochastic processes indexed b)' classes of kernels. in spaces of bounded
functions as discusscd in Chaptcr 18.
PR08LE:\'IS
1. Dc:rivc: the asymptotic diMribution of G;"i'.v meu" tliJl"'1't!/l('t!. which is dclincd as (2)-1 LLi <)
IX, - XJI.
2. Dcrive the projcction of the S3mpJe variancc.
3. Find a kcrncl forthe paramctcrO = E(X - EX) .
4. Find a kcmd for the p;lramctcr O = co\'(X. }'). Show Ihal the corrcspomling U-slatistic is the
sampJc co\'ariance L::'=I(XI - X)(Y; - Y)!CII - I).
5. Findmelimitdi tributionofU =(;)-1 LLi<j(Y' - Y,)(X J -X,).
6. Let Un I and {J,,2 bc U -slalistics with kcrncls" I .md II'!.. rc pcctivcly. Dcrive the joint asymptotic
distrihution of (Uni. Un!)'
7. SUpJX)SC EXi < 00. Dcri\'c the asyrnptutic distrihution of the sCLJucncc ,,-I L L' J X;Xj.
Can you givc a t\\'O line proofwithout using the U -Matistic theorcm'! \Vhat happcns ir EX I = O'!
8. (Mann's h.'St agahl. t trend.) To tc:s.lthc: nuli hyp01hc:sis that a sample XI_ . . . . X" is i.Ld. u ainst
the alternati....e hypothcsis lh;it ahc di tribUli()ns of the X; arc Moch;.stic;llly incrcasing in ;. 1ann
172
U -Statistics
suggested to reject the null hypothesis if the number of pairs (Xi, X j) with i < j and Xi < X j
is large. How can we choose the critical value for large n?
9. Show that the U -statistic U with kemeli {XI + X2 > O}, the signed rank statistic W+, and the
positive-sign statistic S == L7=11 {Xi> O} are related by W+ == ( )U + S in the case that there
are no tied observations.
10. A V-statistic of order 2 is of the form n-2'L7=ILJ=lh(Xi, Xj) where h(x, y) is symmetric
in x and y. Assume that Eh 2 (X 1, XI) < 00 and Eh 2 (X 1, X2) < 00. Obtain the asymptotic
distribution of a V -statistic from the corresponding result for a U -statistic.
11. Define a V -statistic of general order r and give conditions for its asymptotic normality.
12. Derive the asymptotic distribution of n(S - f.-L2) in the case that f.-L4 == f.-L by using the delta-
method (see Example 12.12). Does it make a difference whether we divide by n or n-I ?
13. For any (n x c) matrix ai} we have
n
L ai l,l. ..aic,c == L Il (-1)IBI-l(BI-1)!L Il ai).
i B BEB i=1 JEB
Here the sum on the left ranges over all ordered subsets (i 1, . . . , ic) of different integers from
{ 1, . . . , n} and the first sum on the right ranges over all partitions B of {I, . . . , c} into nonempty
sets (see Example [131]).
14. Given a sequence of i.i.d. random variables XI, X2, . . ., let An be the er-field generated by all
functions of (XI, X2, . . .) that are symmetric in their first n arguments. Prove that a sequence
U n of U -statistics with a fixed kernel h of order r is a reverse martingale (for n > r) with respect
to the filtration Ar ::) Ar+l ::) . . '.
15. (Strong law.) If Elh(Xl, "., Xr)1 < 00, then the sequence Un of U-statistics with kemel h
converges almost surely to Eh(Xl, . . . , X r ). (For r > 1 the condition is not necessary, but a
simple necessary and sufficient condition appears to be unknown.) Prove this. (Use the preceding
problem, the martingale convergence theorem, and the Hewitt-Savage 0-1 law.)
13
Rank, Sign, and Permutation Statistics
Statistics that depend on the observations only through their ranks can be
us ed to test hypotheses on depa rtu res from the null hypothesis that the ob-
servations are identieally distributed. Such rank statisties are attractive,
because they are distribution-free under the nuli hypo the s is and ne ed not
be less efficient, asymptotically. In the case of a sample from a symmetric
distribution, statistics based on the ranks of the absolute values and the
signs of the observations have a similar property. Rank statistics are a
special example of permutation statisties.
13.1 Rank Statistics
The order statisties X N (1) < X N (2) < ... < XN(N) of a set of real-valued observations
XI, . . . , X N i th order statistic are the values of the observations positioned in increasing
order. The rank R N i of Xi among XI, . . . , X N is its position number in the order statistics.
More precisely, if XI, . . . , X N are all different, then RNi is defined by the equation
Xi == XN(R Ni ).
If Xi is tied with some other observations, this definition is invalid. Then the rank RNi is
defined as the average of all indices j such that Xi == XN(j) (sometimes called the midrank),
or altematively as 'L7=I 1 {X j < Xi} (which is something like an uprank).
In this section it is assumed that the random variables XI, . . . , X N have continuous
distribution functions, so that ties in the observations occur with probability zero. We shaH
neglect the latter nulI set. The ranks and order statistics are written with double subscripts,
because N varies and we shaH consider order statistics of samples of different sizes. The
vectors of order statistics and ranks are abbreviated to XN() and R N , respectively.
A rank statistic is any function of the ranks. A linear rank statistic is a rank statistic of
the special form 'L I aN (i, R Ni ) for a given (N x N) matrix (aN (i, j) ). In this chapter
we are be concerned with the subclass of simple linear rank statistics, which take the form
N
L cNi aN,R Ni .
i=I
Here (eNI, . . . , CNN) and (aNI, . . . , aNN) are given vectors in IR N and are called the coeffi-
cients and scores, respectively. The class of simple line ar rank statistics is sufficiently large
173
174
Rcmlc. Sign. (md Pamll/tlTilm SWt;.IiTic'.f
to contain interesting statistics for testing a variety of hypothcscs. In particular. we shali
sec that it contains all "Iocally most powerful" rank statistics. which in another chapter are
shown to he asymptotically cfficient within the cJass of all tests.
Some elemcntary propcnies of ranks and order statistics arc gathercd in the folJowing
lemma.
13.1 Lemma. ut XI. . . . , X N be aram/om :rample fivm a C(mrimlOlIS distriblllimlfimc-
tioll F with dellsiry f. ThclI
(i) the \'ectors XNt) and RN are illdepelldem:
(ii) the \'t?clor X NeJ !las dell.rit)' N! nf::1 f(Xi) (}Il the .rel XI < . .. < .fNo'
(iii) Ihe ,.ariable X NCH has densil.\' N( :l) F(x)'-I(I - F(X»)N-11(:<);for F tlze IIIU'.
for", di.5triblllion 011 [O. I J. il has mean i leN + I) and \'ariance i(N - i + 1)/
(eN + l)l(N + 2»):
(i,,) the ,.tcWr Rh' is IIllifoTml)' distribllled ollllze seI (if all N! pt'rmUllll;lJII.'i of I. 2. . . . .
N:
(\1) for all)' staristie T and penllllrarion r = (r" . . . . r,loJ) of I. 2. .. . , N.
E(T(X r, . . , , X.v) I Rh' = r) = ET(X,"'cr,J' . .. . X''''frr\' );
( 'i) formI>' simple Iinear rallk .flafi.ffit: T = Ei I£"N;ClN.Il,""
ET = NCNU,v:
J _.. _,
\'arT = - (CN, - c.vt L- (aN, - ao\'t.
N - I 1=1 ;....1
Proof.. Statements (i) through (iv) are weJl-known and clementar)'. For the proof of (v). it
is helpful to write T(X I. . . . , X N) as a function of the ranks and the ordcr statistics. Ncxt.
we appJy (i), For the proof of statement (vi), we use that the distributions of the variablcs R.w
and the \'cctors (RNi. R,v,) for i i= j arc uniform on the sets I = (I. . . . . N) and {(i. j) E
{1.: i i: j}. rcspccriveJy. Furthermorc. a double sum of the form Lit:i(b, - b)(b; - h) is
cqual to - E,(b i - b)", .
It follows that rank statistics arc distr;!JlIl;oll.!ree over the set of aJI models in which the
observations arc independent and idcntically distributcd. On the one hand. this makes them
Matislically useless in situations in which the obscrvations are. indeed. arandom sample
from so me distribution. On the other hand, it makcs them of great interesI to dctect cenain
differences indistribution bctwecn the obscrvations. such as in the two-sample problem. ...
the nulJ hypothesis is taken to assert that the observations are identically distributed. then
the critical values for a rank test can be chosen in such away that the probability of an
crror of the first kind is equal to a gi\'en Icvcl a. for any probability distribution in the nuli
hypothcsis. Somewhat surprisingly. this gain is not neccssarily counternctcd by a loss in
as)'mptotic cfficicncy. as we sec in Chapter 14.
13.2 Examp/e (Two- 'amp/e location problem). Supposc that the totaJ set of obser\"ations
consists of two independent random samples. inconsistenlly with the prcceding notation
\",rinen as X" . . . , X m and YI. . . .. Y". SCI N = m + JI and Ic=t RN be the rank \'cclor of
the pooletJ saml'le XI. . . . , X"" YI. . . .. Y n .
13.1 Rallk Stat;.ti/ic.'"
175
\Ve arc interested in testing the nuli hypothesis that the two samples arc idcnticaIJy dis-
tributed (according to a continuous distribution) against the altcrnativc that the distribution
of the second sample is stochastically larger than the distribution of the first sample. Even
without amore precise description of the alternative hypothesis. we can discuss a collec-
tion of useful rank stutistics. If the Y, arc a sample from a stochastically larger distribu.
tion. then the ranks of the Y J in the pooled sample should he rclati\'cly large. Thus. any
mca.. ure of the size of the ranks RN.m+' . . . . . RN N can he uscd as a test statistic. It will bc
distribution.free under the nutl hypothcsis.
The most popular choice in this problem is the Wi/coxml ...lali.'ilic
N
W = L R N ,.
;=",.. ,
This is a simple linear rank statistic with coemcients c = (O, . . . . O, I. . . ., I). and scores
a = (I. ... ,N). The nuli hypothesis is rejected for large values of the \Vilcoxon statistic.
(The \Vilcoxon statistic is cquivalcnt to the ftlal1n- \Vllillle)' Slali. vlk U = L,.) I (Xi YJ}
in that V = U + 1l(1l + I).)
Therc are many other rcasonable choiccs of mnk statistics. some of which are of speciai
interest and havc names. For instance, the ,'all d('r \ Ierde" .fjlClf;Jtic: is defined as
,\'
L <1>-1 (R.\,,).
,-=m + J
Here <1>-1 is the standard norma I quantile function. We shali scc ahcad that this statistic is
particularly nttractive ir it is believed that the undcrl)'ing distribution of the obsel"\',uions
is approximately normal. A general mcthod to generate uscful mnk statistics is discussed
below. O
A critical \'alue for a test based on a (distribution.frce) rank statistic can he found by
simply tabulating its nuli distribution. For a largc numbcr of obscr\',ltions this is a bit tcdious.
In most cases it is also unncccssary. bc.."Cause there exist accuratc asymptotic approximations.
The remainder of this scction is conccrnc:d with pco\'ing asymptotic normality of simpl
line r rank statistics undcr the nuli hypothesis. Apart from bcinguscful for findingcritical
\'aJucs. the thcorem is used subscqucntly to study the ilsymptotic cfliciency of rank tcsts.
Considcr a r U1k statistic orthe fonn T.v = L IC-"N,lIN.Il,"r' For a sequence of this t)'pe to
a5ymptotic 1lly normal. some rcstrictions on the cocnicicnts (' .md scores a are necessary.
In most cases of interest. the scores arc Ugcncratcd" through a gi".en function t/J : lO. I J R
in one of two w,.)'s. Either
(IN, = &/>(UNC,,).
( 13.3)
whcre U NeJlt . . . . U N4N. arc the ordcr statistics of a sample of size N from the 'unifonn
distribution on (O. I); or
<IN. = q,( N I ).
For \....cll.bchavcd functions eP. the se detinitions ,Ire closely rclated and almosl identical.
becausc i/(N + 1) = EUN(,.' Scorcs of the firsl type correspond to the 10caJly mosl
(13.4)
176
Rank, Sign, and Permutation Statistics
powerful rank tests that are discussed ahead; scores of the second type are attractive in view
of their simplicity.
13.5 Theorem. Let RN be the rank vector of an i.i.d. sample XI, . . . , X N from the
continuous distributionfunction F. Let the scores aN be generated according to (13.3) for
a measurable function ep that is not constant almost everywhere, and satisfies Jo 1 ep2 (u) du <
00. Define the variables
N
TN == L CNiaN,R Ni ,
i=1
N
TN == NCN a N + L (CNi - cN)ep(F(X i )).
i=1
Then the sequences T N and T N are asymptotically equivalent in the sense that ET N == ET N
and var (T N - T N) Ivar T N O. The same is true if the scores are generated according to
( 13.4) for a function ep that is continuous almost everywhere, is non constant, and satisfies
N-I L 1 ep2(i I(N + 1)) J ep2(u) du < 00.
Proof. Set U i == F(X i ), and view the rank vector RN as the ranks of the first N elements
of the infinite sequence Ul, U 2, . . .. In view of statement (v) of the Lemma 13.1 the
definition (13.3) is equivalent to
aN,R Ni == E(ep(U i ) I RN).
This immediately yields that the projection of T N onto the set of all square- integrable
functions of RN is equal to T N == E(T N I R N ). It is straightforward to compute that
var T N
var T N
1/(N - 1) L(CNi - CN)2 L(aNi - aN)2
L(CNi - cN)2varep(Ul)
N varaN,RNl
N-I var ep (Ul)
If it can be shown that the right side converges to 1, then the sequences T N and T N are
asymptotically equivalent by the projection theorem, Theorem 11.2, and the proof for the
scores (13.3) is complete.
Using a martingale convergence theorem, we shall show the stronger statement
E(aN,R N1 - ep(U l ))2 O.
(13.6)
Because each rank vector R j -1 is a function of the next rank vec tor R j (for one observation
more), it follows that aN,RNl == E( ep (Ul) IRI, . . . , R N ) almost surely. Because ep is square-
integrable, a martingale convergence theorem (e.g., Theorem 10.5.4 in [42]) yields that the
sequence aN,R N1 converges in second mean and almost surely to E(ep(U l ) I R 1 , R 2 , . . .) . If
ep (U I) is measurable with respect to the a - field generated by R I, R 2 , . . . , then the condi-
tional expectation reduces to ep (Ul) and (13.6) follows.
The projection of Ul onto the set of measurable functions of R N1 equals the conditional
expectation E(U I IRNI) == RNl/(N + 1). By a straightforward calculation, the sequence
var (RNI I (N + 1) ) converges to 1 112 == var Ul. By the projection Theorem 11.2 it follows
that R N1 /(N + 1) Ul in quadratic mean. Because R N1 is measurable in the a-field
generated by R I, R 2 , . . . , for every N, so must be its limit Ul. This concludes the proof
that ep (U I) is measurable with respect to the a - field generated by R 1, R 2 , . . . and hence the
proof of the theorem for the scores 13.3.
13.1 Rank Statistics
177
Next, consider the case that the scores are generated by (13.4). To avoid confusion, write
these scores as b Ni == ep(l/ (N + 1)), and let aNi be defined by (13.3) as before. We shall
prove that the sequences of rank statistics S N and T N defined from the scores aN and b N,
respectively, are asymptotically equivalent.
Because RNl/(N + 1) converges in probability to Ul and ep is continuous almost ev-
erywhere, it follows that q;(RNl/(N + 1)) --+ q;(U l ). The assumption on ep is exactly
that Eq;2(R Nl /(N + 1)) converges to Eep2(U l ). By Proposition 2.29, we conclude that
ep(RNl/(N + 1)) --+ ep(U l ) in second mean. Combining this with (13.6), we obtain that
i/ aNi -b Ni )2 =E(aN,RNl -CP( ::\ ) r O.
By the formula for the variance of a linear rank statistic, we obtain that
N ( - ) 2
var(SN - T N ) Li=l aNi - b Ni - ( a N - b N )
== N --+ O,
var TN Li=l (aNi - a N)2
because var aN, RNl --+ var ep (Ul) > O. This implies that var S N / var T N --+ 1. The proof is
complete. .
U nder the conditions of the preceding theorem, the sequence of rank statistics Le N i aN, RNi
is asymptotically equivalent to a sum of independent variables. This sum is asymptotically
normal under the Lindeberg-Feller condition, given in Proposition 2.27. In the present case,
because the variables q; (F (Xi)) are independent and identically distributed, this is implied
by
maxl::Si::sN (e Ni - C N )2
N --+ O.
Li=l (eNi - CN)2
This is satisfied by the most important choices of vectors of coefficients.
(13.7)
13.8 Corollary. If the veetor of coefficients eN satis.fies (13.7), and the seores are genera-
ted according to (13.3) for a measurable, noneonstant, square-integrable function ep, then
the sequenee of standardized rank statistics (T N - ET N) / sd T N converges weakly to an
N(O, l)-distribution. The same is true ifthe seores are generated by (13.4)for afunction
q; that is eontinuous almost everywhere, is noneonstant, and satisfies N-I L l ep2 (i / (N +
1)) 101 q;2(u) du.
13.9 Example (Monotone score generating functions). Any nondecreasing, nonconstant
function ep satisfies the conditions imposed on score-generating functions of the type (13.4)
in the preceding theorem and corollary. The same is true for every ep that is of bounded
variation, because any such ep is a difference of two monotone functions.
To see this, we recall from the preceding proofthat it is always true that RNl / (N + 1) --+
Ul, almost surely. Furthermore,
( R ) 1 N ( i ) N + 1 N j U+l)/(N+l)
Eq;2 NI == - L q;2 < L q;2(u) du.
N + 1 N i=l N + 1 N i=l i/(N+l)
The right side converges to I cjJ2(u) du. Because ep is continuous almost everywhere, it
follows by Proposition 2.29 that cjJ(RNl/(N + 1)) q;(U l ) in quadratic mean. D
178
Rank, Sign, and Permutation Statistics
13.10 Example (Two-sample problem). In a two-sample problem, in which the first m
observations constitute the first sample and the remaining observations n == N - m the
second, the coefficients are usually chosen to be
CNi = {
i == 1,..., m
i == m + 1, . . . , m + n.
In this case CN == ni N and 'L[:1 (CNi -CN)2 == mnl N. The Lindeberg conditionis satisfied
provided both m ---+ 00 and n ---+ 00. O
13.11 Example (Wilcoxon test). The function ep (u) == u generates the scores aNi ==
i I (N + 1). Combined with "two-sample coefficients," it yields a multiple of the Wilcoxon
statistic. According to the preceding theorem, the sequence of Wilcoxon statistics W N ==
'L[:m+l RNil(N + 1) is asymptotically equivalent to
m n 1
- n" m" n
W N == -- F(Xi) + - F(Yj) + N--.
N i=1 N j=1 N 2
The expectations and variances of these statistics are given by E W N
varW N == mnl(12(N + 1)), and varW N == mnl(12N). D
EW N
n12,
13.12 Example (Median test). The median test is a two-sample rank test with scores of the
form aNi == ep (i I (N + 1) ) generated by the function ep (u) == 1 (O, 1/2] (u). The corresponding
test statistic is
N { N+1 }
. L 1 RNi < 2 .
l=m+l
This counts the number of Y j less than the median of the pooled sample. Large values of
this test statistic indicate that the distribution of the second sample is stochastically smaller
than the distribution of the first sample. D
The examples of rank statistics discussed so far have a direct intuitive meaning as statistics
measuring a difference in location. It is not always obvious to find a rank statistic appropriate
for testing certain hypotheses. Which rank statistics measure a difference in scale, for
instance?
A general method of generating rank statistics for a specific situation is as follows.
Suppose that it is required to test the null hypothesis that XI, . . . , X N are i.i.d. versus the
alternative that XI, . . . , X N are independent with Xi having a distribution with density JCNi B ,
for a given one-dimensional parametric model e 1--+ JB. According to the Neyman-Pearson
lemma, the most powerful rank test for testing Ho : e == O against the simple alternative
HI : e == e rejects the null hypothesis for large values of the quotient
P ( R - r )
B N - - N , P (R - )
-.B N-r.
PO(RN == r)
Equivalently, the null hypothesis is rejected for large values of P B (R N == r). This test
depends on the alternative e, but this dependence disappears if we restrict ourselves to
13.1 Rank Statistics
179
alternative s e that are sufficiently close to O. lndeed, under regularity conditions,
P()(R N == r) - PO(RN == r)
= f... iN=r ([VCNiĐ (Xi) - D JO(Xi)) dXl . . . dXN
N . N
== e f . .. ( L CNi Jo (Xi) UJO(Xi) dXl ... dXN + oce)
J RN=r i=l Ja i=l
1 N ( j )
== e, L CNiEo -2. (Xi) I RN == r + oce).
N. i=l Ja
Conclude that, for small e > O, large values of P() (R N == r) correspond to large values of
the simple linear rank statistic T N == L{:l CNiaN,RNi, for the vector aN of scores given by
. .
aNi = Eo j (XN(iJ = E j (Fo-l (UN(iJ).
These scores are of the form (13.3), with score-generating function ep == (j 0/ Ja) o FO-l.
Thus the corresponding rank statistics are asymptotically equivalent to the statistics
N .
Li=l CNi J o/Ja (Xi)'
Rank statistics with scores generated as in the preceding display yield locally most pow-
erful rank tests. They are most powerful within the class of all rank tests, uniformly in a
sufficiently small neighbourhood (0,8) of O. (For a precise statement, see problem 13.1).
Such a local optimality property may seem weak, but it is actually of considerable im-
portance, particularly if the number of observations is large. In the latter situation, any
reasonable test can discriminate well between the nulI hypothesis and "distant" alterna-
tives. A good test proves itself by having high power in discriminating "cio se" alternatives.
13.13 Example (Two-sample scale). To generate a test statistic for the two-sample scale
problem, let J() (x) == e-() J (e-() x) for a fixed density J. lf Xi has density JCNi() and the
vector C is chosen equal to the usual vector of two-sample coefficients, then the first m
observations have density Ja == J; the last n == N - m observations have density J(). The
alternative hypothesis that the second sample has larger scale corresponds to e > O. The
scores for the locally most powerful rank test are given by
aNi = -E(l + F-l(UN(iJ (F- 1 (UN(i)))).
For instance, for J equal to the standard normal density this leads to the ..rank statistic
L[:m+l aN,RNi with scores
aNi == E<I>-1(U N (i))2 - 1.
The same test is found for J equal to a normal density with a different mean or variance.
This follows by direct calculation, or alternatively from the fact that rank statistics are
location and scale invariant. The latter implies that the probabilities P I-L,a,() (R N == r) of
the rank vector RN of a sample of independent variables XI, . . . , X N with Xi distributed
according to e-() J (e-() (x - M) / a ) / a do not depend on (M, a). Thus the procedure to
generate locally most powerful scores yields the same result for any (M, a). D
180
Rank, Sign, and Permutation Statistics
13.14 Example (Two-sample location). In order to find locally most powerful tests for
location, we choose Je (x) == J (x - e) for a fixed density J and the coefficients c equal to
the two-sample coefficients. Then the first m observations have density J (x) and the last
n == N - m observations have density J(x - e). The scores for a locally most powerful
rank test are
aNi = -E( j (p-I (UN(iJ) ).
For the standard normal density, this leads to a variation of the van der Waerden statistic.
The Wilcoxon statistic corresponds to the logistic density. D
13.15 Example (Log rank test). The cumulative hazardJunction corresponding to a con-
tinuous distribution function F is the function A == -log( 1 - F). This is an important
modeling tool in survival analysis. Suppose that we wish to test the null hypothesis that two
samples with cumulative hazard functions A x and A y are identically distributed against
the alternative that they are not. The hypothesis of proportional hazards postulates that
A y == e A x for a constant e, meaning that the second sample is a factor e more "at risk" at
any time. If we wish to have large power against altematives that satisfy this postulate, then
it makes sense to use the locally most powerful scores corresponding to a family defined
by Ae == e A 1. The corresponding family of cumulative distribution functions Fe satisfies
1 - Fe == (1 - F 1 )e and is known as the family of Lehmann alternatives. The locally most
powerful scores for this family correspond to the generating function
a a
cjJ(u) == -log -(1 - F e )(x) l e-l x-F-l ( u ) == 1 -log(l - u).
ae ax - , - 1
It is fortunate that the score- generating function does not depend on the baseline hazard
function Al, The resulting test is known as the log rank test. The test is related to the
Savage test, which uses the scores
N 1 ( i )
aN,i == L -:- -log 1 - .
j=N-i+l ] N + 1
The log rank test is a very popular test in survival analysis. Then usually it needs to be
extended to the situation that the observations are censored. D
13.16 Example (More-sample problem). Suppose the problem is to test the hypothesis
that k independent random samples XI, ..., X NJ , X N1 + 1 , ..., X N2 , .. ., X Nk _ 1 + 1 , ..., X Nk
are identical in distribution. Let N == Nk be the total number of observations, and let RN be
the rank vector of the pooled sample XI, . . . , X N. Given scores aN inference can be based
on the rank statistics
NJ
T N1 == LaN,R Ni ,
i=1
N2 Nk
T N2 == L aN,RNi'.'.' TNk == L aN,R Ni .
i=N]+1 i=Nk-l+1
The testing procedure can consist of several two-sample tests, comparing pairs of (pooled)
subsamples, or on an overall statistic. One possibility for an over all statistic is the chi -square
13.2 Signed Rank Statistics
181
statistic. For nj == Nj - N j - l equal to the number of observations in the jth sample, define
2 (TNj-nj a N)2
C N == .
j=l n jvar ep (Ul)
If the scores are generated by (13.3) or (13.4) and all sample sizes nj tend to infinity, then
every sequence T Nj is asymptotically normal under the nulI hypothesis, under the conditions
of Theorem 13.5. In fact, because the approximations t Nj are joint1y asymptotically normal
by the multivariate central limit theorem, the vec tor T N == (T NI, . . . , T Nk) is asymptotically
normal as well. By elementary calculations, if ni / N -+ Ai,
)"1 (1 - Al) -AlA2 -AlAk
TN - ETN -A2 A l A2(1 - A2) -A2 A k
sdcp(U l ) "v'-7 Nk O,
-AkAl -Ak A 2 Ak(l - Ak)
This limit distribution is similar to the limit distribution of a sequence of multinomial vectors.
Analogously to the situation in the case of Pearson's ehi-square tests for a multinomial
distribution (see Chapter 17), the sequence C converges in distribution to a ehi-square
distribution with k - 1 degrees of freedom.
There are many reasonable choices of scores. The most popular choice is based on
ep (u) == u and leads to the Kruskal- Wallis test. Its test statistic is usually written in the form
12 k ( _ N + 1 ) 2
N(N - l) f;n j Rj. - 2 '
",Nj R .
i=Nj-l+l Nz
R j. ==
n.
]
This test statistic measures the distance of the average scores of the k samples to the average
score (N + 1)/2 of the pooled sample.
An alternative is to use locally asymptotically powerful scores for a family of distribu-
tions of interest. Also, choosing the same score generating function for all subsamples is
convenient, but not necessary, provided the ehi -square statistic is modified. D
13.2 Signed Rank Statistics
The sign of a number x, denoted sign(x), is defined to be -1, O, or 1 if x < O, x == O or
x > O, respectively. The absolute rank Rti of an observation Xi in a sample XI, . . . , X N
is defined as the rank of IX i I in the sample of absolute values IX 11, . . . , IX N I. A simple
linear signed rank statistic has the form
N
" aN R+. sign(X i ).
' NI
i=l
The ordinary ranks of a sample can always be derived from the combined set of absolute
ranks and signs. Thus, the vectors of absolute ranks and signs are together statistically more
informative than the ordinary ranks. The difference is dramatic if testing the location of a
symmetric density of a given form, in which case the class of signed rank statistics contains
asymptotically efficient test statistics in great generality.
182
Rank, Sign, and Permutation Statistics
The main attraction of signed rank statistics is their simplicity, particularly their being
distribution-free over the set of all symmetric distributions. Write IX I, R , and signN(X)
for the vectors of absolute values, absolute ranks, and signs.
13.17 Lemma. Let XI, . . . , X N be arandom sample from a continuous distribution that
is symmetric about zero. Then
(i) the vectors (IXI, R ) and signN(X) are independent;
(ii) the vector R is uniformly distributed over {I, . . . , N};
(iii) the vector sign N (X) is uniformly distributed over {-I, l}N;
(iv) for any signed rank statistic, var 'Lt:I aN,Rti sign(X i ) == 'L =I a i'
Consequently, for testing the nulI hypothesis that a sample is i.i.d. from a continuous,
symmetric distribution, the criticallevel of a signed rank statistic can be set without further
knowledge of the "shape" of the underlying distribution.
The nulI hypothesis of symmetry arises naturally in the two-sample problem with paired
observations. Suppose that, given independent observations (XI, YI), . . . , (X N , Y N), it is
desired to test the hypothesis that the distribution of Xi - Yi is "centered at zero." If the
observations (Xi, Yi) are exchangeable, that is, the pairs (Xi, Yi) and (Yi, Xi) are equal
in distribution, then Xi - Yi is symmetrically distributed about zero. This is the case, for
instance, if, given a third variable (usually calIed "factor"), the observations Xi and Yi are
conditionalIy independent and identically distributed. For the vector of absolute ranks to
be uniformly distributed on the set of all permutations it is necessary to assume in addition
that the differences are identically distributed.
For the signs alone to be di s tribution- free, it suffices, of course, that the pairs are inde-
pendent and that P(X i < Yi) == P(X i > Yi) == i for every i. Consequently, tests based on
only the signs have a wider applicability than the more general signed rank tests. However,
depending on the model they may be less efficient.
13.18 Theorem. Let XI, . . ., X N be arandom sample from a continuous distribution
that is symmetric about zero. Let the scores aN be generated according to (13.3) for a
measurable function ep such that Jo I ep2 (u) du < 00. For F+ the distribution function of
IXII, dejine
N
TN == " aN R+. sign(X i ),
' Nz
i=I
N
TN == L ep(F+ (IX i I)) sign(X i ).
i=I
Then the sequences TN and TN are asymptotically equivalent in the sense that N-Ivar (T N -
T N ) ---+ O. Consequently, the sequence N- I j 2 T N is asymptotically normal with mean zero
and variance Jo I ep2(u) du. The same is true ifthe scores are generated according to (13.4)
for a function ep that is continuous almost everywhere and satisfies N-I 'Lt:I cjJ2 (i / (N +
1)) ---+ Jo I ep2 (u) du < 00.
Proof. Because the vec tor s signN(X) and (IXI, R ) are independent and E signN(X) ==
O, the means of both T N and T N are zero. Furthermore, by the independence and the
orthogonality of the signs,
E(T N - TN)2 = NE(aN,R 1 - 4>(F+(lX 1 1))t
13.4 Rank Statistics for Independence
183
The expectation on the right side is exactly the expression in (13.6), evaluated for the
special choice Ul == F+ (I XII). This can be shown to converge to zero as in the proof of
Theorem 13.5. .
13.19 Example (Wilcoxon signed rank statistic). The Wilcoxon signed rank statlstlc
W N == "2:/:1 Rtž sign(X i ) is obtained from the score-generating function <p(u) == u. Large
values of this statistic indicate that large absolute values I Xi 1 tend to go together with pos-
itive Xi. Thus large values of the Wilcoxon statistic suggest that the location of the Xi is
larger than zero. Under the nulI hypothesis that XI, . . . , X N are i.i.d. and symmetricalIy
distributed about zero, the sequence N- 3 / 2 W N is asymptoticalIy normal N(O, 1/3). The
variance of W N is equal to N(2N + l)(N + 1)/6.
The signed rank statistic is asymptoticalIy equivalent to the U -statistic with kemel
h(Xl, X2) == I{Xl + X2 > O}. (See problem 12.9.) This connection yields the limit distri-
bution also under nonsymmetric distributions. O
Signed rank statistics that are localIy most powerful can be obtained in a similar fashion
as localIy most powerful rank statistics were obtained in the previous section. Let f be
a symmetric density, and let XI, . . . , X N be arandom sample from the density f (. -8).
Then, under regularity conditions,
Pe(signN(X) == s, Rt == r) - Po (sign N (X) == s, Rt == r)
N f'
== -8 Eo L sign(X i ) - (IX i I) {signN(x) == s, Rt == r} + oce)
i=1 f
1 N ( f' )
== -8 N , L SiEO - (IX i I) I Rti == ri + oce).
2 N. i=1 f
In the second equality it is used that f' / f (x) is equal to sign(x) f' / f (Ix I) by the skew
symmetry of f' / f. It folIows that locally most powerJul signed rank statistics for testing
J against J(. -e) are obtained from the scores
aNi = _E f' ((p+)-l(U N (i))).
f
These scores are of the form (13.3) with score-generating function <p == -(f' / f) o (F+)-I,
whence localIy most powerful rank statistics are asymptoticalIy linear by Theorem 13.18.
By the symmetry of F, we have (p+)-1 (u) == p-I (u + 1)/2).
13.20 Example. The Laplace density has score function f' / f (x) == sign(x) == 1, for
X > O. This leads to the localIy most powerful scores aNi - 1. The corresponding test
statistic is the sign statistic T N == 'L[:1 sign(X i ). Is it surprising that this simple statistic
possesses an optimality property? It is shown to be asymptoticalIy optimal for testing
Ho : e == O in Chapter 15. O
13.21 Example. The locally most powerful score for the normal distribution are aNi ==
E<I:>-1 (UN(i) + 1)/2). These are appropriately known as the normal (absolute) scores. O
184
Rank, Sign, and Permutation Statistics
13.3 Rank Statistics for Independence
Let (XI, YI), . . . , (X N, Y N) be independent, identicalIy distributed bivariate vectors, with
continuous marginal distributions. The problem is to determine whether, within each pair,
X i and Yi are independent.
Let RN and S N be the rank vectors of the samples XI, . . . , X N and YI, . . . , Y N, respec-
tively. If Xi and Yi are positively dependent, then we expect the vectors RN and SN to be
roughly parallel. Therefore, rank statistics of the form
N
L aN,RNibN,SNi'
i=l
with aN and b N increasing vectors, are reasonable choices for testing independence.
Under the null hypothesis of independence of Xi and Yi, the vectors RN and SN are
independent and both uniformly distributed on the permutations of {I, . . . , N}. Let R be
the vec tor of ranks of XI, . . . , X N if first the pairs (XI, YI), . . . , (X N , Y N) have been put in
increasing order of YI < Y 2 < . . . < Y N. The coordinates of R are calIed the antiranks.
Under the null hypothesis, the antiranks are also uniformly distributed on the permutations
of {I, . . . , N}. By the detinition of the antiranks,
N N
'""" aN R N ' bN S N ' == '""" aN RO, b Ni .
' z , z 'Nz
i=1 i=1
The right side is a simple linear rank statistic and can be shown to be asymptotically normal
by Theorem 13.5.
13.22 Example (Spearman rank corre latio n ). The simplest choice of scores corresponds
to the generating function <p (u) == u. This leads to the rank co rre [ation coefficient p N , which
is the ordinary sample correlation coefticient of the rank vectors RN and SN. Indeed, be-
cause the rank vectors are permutations of the numbers 1, 2, . . . , N, their sample mean and
variance are tixed, at (N + 1)/2 and N(N + 1)/12, respectively, and hence
N - -
Li=I(R Ni - RN)(SNi - SN)
PN == ( N - N - ) 1/2
Li=1 (R Ni - RN)2 Li=1 (SNi - SN)2
12 N N + 1
- '""" R N ,S N ' - 3
- N(N - l)(N + 1) b l l N - l'
Thus the tests based on the rank correlation coefticient p N are equivalent to tests based on
the signed rank statistic L R N i S N i .
It is straightforward to derive from Theorem 13.5 that the sequence -JR PN is asymptot-
icalIy standard normal under the nulI hypothesis of independence. D
*13.4 Rank Statistics under Alternatives
Let RN be the rank vec tor of the independent random variables XI, . . . , X N with continu-
ous distribution functions F 1 , . . . , F N. Theorem 13.5 gives the asymptotic distribution of
simple, linear rank statistics under very mild conditions on the score-generating function,
13.4 Rank Statistics under Alternatives
185
but under the strong assumption that the distribution functions F i are all equal. This is suffi-
cient for setting critical values of rank tests for the nulI hypothesis of identical distributions,
but for studying their asymptotic efficiency we also need the asymptotic behavior under
altematives. For instance, in the two-sample problem we are interested in the asymptotic
distributions under alternative s of the form F, . . . , F, G, . . . , G, where F and G are the
distributions of the two samples .
For alternative s that converge to the null hypothesis "sufficiently fast," the best approach
is to use Le Cam' s third lemma. In particular, if the log likelihood ratios of the alternative s
Fn, . . . , Fn, G n , . . . , G n with respect to the null distributions F, . . . , F, F, . . . , F allow
an asymptotic approximation by a sum of the type L li (Xi), then the joint asymptotic
distribution of the rank statistics and the log likelihood ratios under the null hypothesis
can be obtained from the multivariate central limit theorem and Slutsky's lemma, because
Theorem 13.5 yields a similar approximation for the rank statistics. Next, we can apply Le
Cam' s third lemma, as in Example 6.7, to find the limit distribution of the rank statistics
under the alternatives. This approach is relatively easy, and is sufficiently general for most
of the questions of interest. See sections 7.5 and 14.1.1 for examples.
More general altematives must be handled directly and appear to require stronger con-
ditions on the score-generating function. One possibility is to write the rank statistic as a
functional of the empirical distribution function JF N, and the weighted empirical distribution
JF (x) == N-I Lt:l CNi 1 {Xi < x} of the observations. Because RNi == NJFN(X i ), we have
1 N J
N L CNiaN,R N ; = aN,NFN(x) dJF (x).
1=1
N ext, we can apply a von Mises analysis, using the convergence of the empirical distribution
functions to Brownian bridges. This method is explained in general in Chapter 20.
In this section we illustrate another method, based on Hajek's projection lemma. To
avoid technical complications, we restrict ourselves to smooth score-generating functions.
- -c . 1 N
Let F N be the average of F 1 , . . . , F N and let F N be the welghted sum N- Li=1 CNi F i , and
define
N ( )
RNi
TN = 8 CNicfJ N + 1 '
TN = t [CNicfJ( F N(Xi)) + l cfJ/( F N(X))d F (X)].
We shall show that the variables t N are the Hajek projections of approximations to the
variables T N, up to centering at mean zero. The Hajek projections of the variables T N
themselves give a better approximation but are more complicated.
13.23 Lemma. If C/J : [O, 1] r+ ffi. is twice continuously differentiable, then there exists a
universal constant K such that
N
A 1" 2 ( /2 112 )
var (T N - T N ) < K - (CNi - CN) 1Ic/J 1100 + 11c/J 1100 .
N i=1
Proof. Because the inequality is for every fixed N, we delete the index N in the proof.
Furthermore, because the assertion concerns a variance and both T N and t N change by a
186
Rank, Sign, and Permutation Statistics
constant if the C Ni are replaced by C Ni - eN, it is not a loss of generality to as sume that
eN == O. (Evaluate the integral defining t N to see this.)
The rank of Xi can be written as R i == 1 + Lki=i 1 {X k < Xi}. This representation and
alittle algebra show that
( Ri ) - 1 I - I 1
E X. - F ( X. ) == 1 - F ( X. ) - F. ( X. ) <-.
N+l l l N+l l l l - N
Furthermore, applying the Marcinkiewitz-Zygmund inequality (e.g., [23, p. 356]) condi-
tionally on Xi, we obtain that
( ) 4
R i -
E N + 1 - F(X i )
= (N 1)4 E ( (1{Xk < X;} - Fk(X;») + 1 - F (X;) _ F;(X;») 4
1 ( 1 4 ) 1 1
< -EE - '""' ( l { X k < X. } - F k( X. )) X. + - < -.
rv N2 N f;t - l l l N4 rv N2
Next, developing ep in a two-term Taylor expansion around F(X i ), for each term in the sum
that defines T, we see that there exist random variables K i that are bounded by II ep" 1100 such
that
T = tc;q;{ F (X;») + t Ci ( RNi - F (Xi) ) ep'( F (X i ))
i=I i=I N + 1
N ( ) 2
RNi -
+ '""' c. - F ( X. ) K.
lN+l l l
l=I
==: To + TI + T 2 .
Using the Cauchy-Schwarz inequality and the fourth-moment bound obtained previously,
we see that the quadratic term T 2 is bounded above in second mean as in the lemma. The
leading term To is a sum of functions of the single variables Xi, and is the first part of t.
We shall show that the linear term TI is asymptotically equivalent to its Hajek projection,
which, moreover, is asymptotically equivalent to the second part of t, up to a constant. The
Hajek projection of TI is equal to, up to a constant,
L C iL E [ N:l q;'{ F (X;») X j ] - Lc; F (X;)q;'{ F (X i »)
l } l
==LCi [ L E [ R; q;'{ F (X i ») X j ]
. '-/.' N + 1
l iTl
+ Ci(E( N: 1 Xi) - F (X;»)q;'{ F (Xi»)'
The second term is bounded in second mean as in the lemma the first term is equal to
1 LCi LE(l{X j < X;}q;'{ F (X i ») IX j ) + constant.
N + 1. '-/.'
l iTl
13.4 Rank Statistics under Alternatives
187
If we replace (N + 1) by N, write out the conditional expectation, add the diagonal terms, and
remove the constant, then we obtain the second term in the definition of T. The difference
between these two expressions is bounded above in second mean as in the lemma.
To conclude the proof it suffices to show that the difference between TI and its Hajek
projection is negligible. We employ the Hoeffding decomposition. Because each of the
variables R i ej;' ( F (X i)) is contained in the space LI A I :::2 HA, the difference between TI
and its Hajek projection is equal to the projection of TI onto the space LIAI=2 HA. This
projection has second moment
1 2 L E ( PA L Ci L I{Xk < Xi}cp'( F (X i ) ) ) 2.
(N + 1) IAI=2 i k
The proj ection of the variable 1 {X k < Xi }cp' ( F (X i) ), which is contained in the space H{k, i},
onto the space H{a,b} is zero unless {a, b} c {k, i}. Thus, the expression in the preceding
display is equal to
(N 1)2 LE(Cb 1 {Xa < Xb}4>'( F (X b )) +C a 1{Xb < X a }4>'( F (X a ))r.
a<b
This is bounded by the upper bound of the lemma, as desired. The proof is complete. .
Asaconsequenceofthelemma, thesequences (T N - ETN )/sd TN and (T N - ETN )/sd TN
have the same limiting distribution (if any) if
L::I (CNi - eN)2 O
A .
N var T N
This condition is certainly satisfied if the observations are identically distributed. Then
the rank vector is uniformly distributed on the permutations, and the explicit expression for
var T N given by Lemma 13.1 shows that the left side (with var T N instead of var T N ) is of the
order 0(1/ N). Because this leaves much too spare, the condition remains satisfied under
small departures from identical distributions, but the general situation requires a calculation.
Under the conditions of the lemma we have the approximation
N ( . ) N
ET N CNL4> l + L(CNi -cN)E4>( F N (X i )).
i=1 N + 1 i=1
The square of the difference is bounded by the upper bound of the lemma.
The preceding lemma is restricted to smooth score-generating functions. One possibility
to extend the result to more general scores is to show that the difference between the rank
statistics of interest and suitable approximations by rank statistics with smooth scores is
smalI. The following lemma is useful for this purpose, although it is suboptimal if the
observations are identically distributed. (For a proof, see Theorem 3.1, in [68].)
13.24 Lemma (Variance inequality). For nondecreasing coefficients aNI <
and arbitrary scores CNI, . . . , CNN,
< aNN
N N
var L CNiaN, RNi < 21 ax (CNi - eN )2L (aNi - a N)2.
. 1 1:::1:::N . 1
1= 1=
188
Rank, Sign, and Permutation Statistics
13.5 Permutation Tests
Rank tests are examples of permutation tests. General permutation tests also possess a
distribution-free level but still use the values of the observations next to their ranks. In this
section we illustrate this for the two-sample problem.
Suppose that the null hypothesis Ha that two independent random samples XI, . . . , X m
and YI, . . . , Y n are identically distributed is rejected for large values of a test statistic
T N (X 1, . . . , X m , YI, . . . , Y n ). Write Z(1), . . . , ZeN) for the values of the pooled sample
stripped of its original order. (N == m + n.) Dnder the null hypothesis each permutation
Z7fl' . . . , ZTrN of the N values is equally likely to lead back to the original observations. More
precisely, the conditional null distribution of XI, . . . , X m , YI, . . . , Y n given Z(1), . . . , ZeN)
is uniform on the N! permutations of the latter sample. Thus, it would be reasonable to
reject Ha if the observed value T N (XI, ..., X m , YI, ..., Yn) is among the 100a% largest
values T N ( Z7f l ' . . . , Z7fN) as n ranges over all permutations. Then we obtain a test of level
a, conditionally given the observed values and hence also unconditionally.
Does this procedure work? Does the test have the desired power? The answer is
affirmative for statistics T N that are sums, in the sense that, asymptotically, the permutation
test is equivalent to the test based on the normal approximation to T N. If the latter test
performs well, then so does the permutation test.
We consider statistics of the form, for a given measurable function f,
1 mIn
T N (X I , ..., X m , YI, ..., Y n ) == - Lf(X i ) - - Lf(Y j ).
m. I n. 1
l= J=
These statistics include, for instance, the score statistics for testing that the two samples
have distributions PO and pe, respectively, for which we take f equal to the score function
Pol PO of the model. Because a permutation test is conditional on the observed values,
and T N is fixed once Lj f (Y j ) and Li f (Zi) are fixed, it would be equivalent to consider
statistics of the form L j f (Y j ) .
Let (nNI, . . . , nN N) be uniformly distributed on the N! permutations of the numbers
1,2, . .., N, and be independent of XI, . . . , X m , YI, . . . , Y n .
13.25 Theorem. Let both Ef2(X I ) and Ef2(y I ) be finite, and suppose that m, n --+ 00
such that ml N --+ 'A E (0,1). Then, given almost every sequence XI, X 2 , .. . , YI, Y2, ...,
the sequence ,JFiT N (ZTrNl' . . . , Z7fNN) is asymptotically normal with mean zero. Under the
null hypothesis the asymptotic variance is equal to var f(X I )/('A(l - 'A)).
Proof. Conditionally on the values of the pooled sample, the statistic N T N (ZnNl' . . . ,
Z7fNN) is distributed as the simple linear rank statistic L{:I CNiaN,RNi with coefficients and
scores
CNi == f(Zi),
! N
m'
aNi == _ N
n '
i < m
l > m
Here R NI , . . . , RN N are the antiranks of JTNI, . . . , nN N defined by the equation L c N,TrNi aNi
== L CNiaN,R Ni (for any numbers CNi and aNi)'
13.5 Permutation Tests
189
The coefficients satisfy relation (13.7) for almost every sequence XI, X 2, . . . , YI, Y 2 , . . . ,
because, by the law of large numbers,
C AEfk(X 1 ) + (1 - A)Efk(y 1 ),
1 2 as
- max CNi --* O.
N 1 <i <N
k==I,2,
The scores are generated as aNi == epN (i / (N + 1)) for the functions
I N
m'
epN(U) == _ N
n '
< m
U - N+l '
m
U > N+l '
These functions depend on N, unlike the situation of Theorem 13.5, but they converge
to the fixed function ep == A -II[o,A) - (1 - A)-II(A,I]' By aminor extension of Theo-
rem 13.5, the sequence L CNiaN,RNi is asymptotically equivalent to L(cNi - CN )ep(U ž ),
for a uniform sample Ul, . . . , U N. The (asymptotic) variance of the latter variable is easy to
compute. .
By the central limit theorem, under the null hypothesis,
-JNT N (X 1 , . .., X m , YI, ..., Y n ) -v--+ N(O, 0- 2 ),
2 var f(X 1 )
o- == .
A(1 - A)
The limit is the same as the conditionallimit distribution of the sequence VN T N (ZnNl' . . . ,
ZnNN) under the null hypothesis. Thus, we have a choice of two sequences of tests, both of
asymptotic level a, rejecting Ha if:
- VNT N (X 1 ,..., X m , YI, ..., Y n ) > ZaO-; or
- VNT N (X 1 ,..., X m , YI, ..., Y n ) > CN(X 1 , ..., X m , YI, ..., Y n ), where
CN (X 1, . . . , X m , YI, . . . , Y n ) is the upper a-quantile of the conditional
distribution of -JNT N (ZnNl' . . . , ZnNN) given Z(I), . . . , Z(N).
The second test is just the permutation test discussed previously. By the preceding theorem
the "random critical values" C N (X 1, . . . , X m , YI, . . . , Y n ) converge in probability to ZaO-
under Ha. Therefore the two tests are asymptotically equivalent under the null hypoth-
esis. Furthermore, this equivalence remain s under "contiguous alternatives" (for which
again CN(X 1 , ..., X m , YI, ..., Y n ) ZaO-; see Chapter 6), and hence the local asymp-
totic power functions as discussed in Chapter 14 are the same for the two sequences of
tests.
The preceding theorem also shows that the sequence of "critical values" eN (X 1, . . . , X m ,
YI, . . . , Y n ) remains bounded in probability under every alternative. Because VNT N
(XI,..., X m , YI,".' Y n ) -v--+ ooifEf(X 1 ) > Ef(Y 1 ), thepowerat any alternative with this
property converges to 1. Thus, permutation tests are an attractive alternative to both rank
and classical tests. Their main drawback is computational complexity. The dependence of
the null distribution on the observed values means that it cannot be tabulated and must be
computed for every new data set.
190
Rank, Sign, and Permutation Statistics
*13.6 Rank Central Limit Theorem
The rank central limit theorem Theorem 13.5, is slightly special in that the scores aNi are
assumed to be of one of the forms (13.3) or (13.4). In this section we record what is com-
monly viewed as the rank central limit theorem. For a proof see [67]. For given coefficients
and scores, let
n
C == L(cNi - CN)2,
i=l
n
A == L(aNi - a N )2.
i=l
13.26 Theorem (Rank central limit theorem). Let TN == L CNžaN,RNi be the simple lin-
ear rank statistic with coefficients and scores such that maxl::Si::SN laNi - a N 1/ AN --+ O
and maXl<i<N ICNi - cNI/CN --+ O, and let the rank vector RN be uniformly distributed
on the set of all N! permutations of {1, 2, . . . , N}. Then the sequence (T N - ETN )jsd TN
converges in distribution to a standard normal distribution if and only if, for every 8 > O,
laNi - a NI 2 1cNi - cNI 2
2 2 --+ o.
ANC N
LL
(i,j): JjVlaNi-aNllcNi-cNI>eANCN
Notes
The classical reference on rank statistics is the book by Hajek and Sidak [71], which stiH
makes wonderful reading and gives extensive references. Its treatment of rank statistics for
nonidenticaHy distributed observations is limited to contiguous alternatives, as in the first
sections of this chapter. The papers [43] and [68] remedied this, shortly after the publication
of the book. Section 13.4 reports only a few of the resuits from these papers, which, as
does the book, use the projection method. An alternative approach to obtaining the limit
distribution of rank statistics, initiated by Chernoff and Savage in the late 1950s and refined
many times, is to write them as functions of empirical measures and next apply the von
Mises method. We discuss examples of this approach in Chapter 20. See [134] for amore
comprehensive treatment and further references.
PROBLEMS
1. This problem asks one to give a precise meaning to the notion of a locally most powerful test.
Let TN be a rank statistic based on the "locally most powerful scores." Let a == PO(TN > caJ for
a given number C a . (Then a is a naturallevel of the test statistic, a level that is attained without
randomization.) Then there exists 8 > O such that the test that rejects the null hypothesis if
T N > C a is most powerful within the class of all rank tests at level a uniformly in the altematives
e E (0,8).
(i) Prove the statement.
(ii) Can the statelnent be extended to arbitrary levels?
2. Find the asymptotic distribution of the median test statistic under the null hypothesis that the two
samples are identically distributed and continuous.
3. Show that ,Jn times Spearman's rank correlation coefficient is asymptotically standard norma!.
4. Find the scores for a locally most powerful two-sample rank test for location for the Laplace
family of densities.
Problems
191
5. Find the scores for a locally most powerful two-sample rank test for location for the Cauchy
family of densities.
6. For which density is the Wilcoxon signed rank statistic locally most powerful?
7. Show that Spearman's rank correlation coefficient is a linear combination of Kendall's T and the
U -statistic with (asyrnmetric) kernel h (x, Y, z) == sign(xI - YI) sign(x2 - Z2). This decompo-
sition yields another method to prove the asymptotic normality.
8. The symmetrized Siegel- Tukey test is a two-sample test with score vec tor of the form aN
(1,3,5, . . . , 5, 3, 1). For which type of alternative hypothesis would you use this test?
9. For any aNi given by (13.3), show that aN == Jo 1 <jJ(u) du.
14
Relative Efficiency of Tests
The quality of sequences of tests can be judged from their power at alter-
natives that become closer and closer to the nulI hypothesis. This moti-
vates the study oflocal asymptotic powerfunctions. The relative efficiency
of two sequences of tests is the quotient of the numbers of observations
needed with the two tests to obtain the same level and power. We discuss
several types of asymptotic relative efficiencies.
14.1 Asymptotic Power Functions
Consider the problem of testing a nulI hypothesis Ha : e E 8 0 versus the alternative HI : e E
8 1 . The power function of a test that rejects the nulI hypothesis if a test statistic falls into a
critical region Kn is the function e Trn (e) == P8 (T n E Kn), which gives the probability
of rejecting the nulI hypothesis. The test is of level cl if its size sup{ Trn (e) : e E 8 0 } does
not exceed cl. A sequence of tests is called asymptoticalIy of leve I cl if
lim sup sup Trn (e) < Cl.
n oo 8EBa
(An alternative detinition is to drop the supremum and require only that lim sup Trn (e) < cl
for every e E 8 0 .) A test with power function Trn is better than a test with power function
!I n if both
Trn(e) < !In(e),
and Tr n ( e) > !I n ( e) ,
e E 8 0 ,
e E 8 1 .
The aim of this chapter is to compare tests asymptotically. We consider sequences of tests
with power functions Trn and !I n and wish to decide which of the sequences is best as
n ---+ 00. Typically, the tests corresponding to a sequence TrI, Tr2, . . . are of the same type.
For instance, they are all based on a certain U -statistic or rank statistic, and only the number
of observations changes with n. Otherwise the comparison would have little relevance.
A tirst idea is to consider limiting power functions of the form
Tr (e) == lim Trn (e).
n oo
If this limit exists for all e, and the same is true for the competing tests !I n, then the se-
quence Trn is better than the sequence !I n if the limiting power function Tr is better than the
192
14.1 Asymptotic Power Functions
193
limiting power function Tr . It turns out that this approach is too naive. The limiting power
functions typically exist, but they are trivial and identical for all reasonable sequences of
tests.
14.1 Example (Sign test). Suppose the observations XI, . . . , X n are arandom sample
from a distribution with unique median e. The null hypothesis Ha : e == O can be tested
against the alternative HI : e > O by means of the sign statistic Sn == n-I :L7=1 1 {Xi> O}.
If FCx -e) is the distribution function of the observations, then the expectation and variance
of Sn are equal to /-Lce) == 1 - FC -e) and a 2 ce)/n == (1 - FC -e) )FC -e)/n, respectively.
By the normal approximation to the binomial distribution, the sequence ,jn(Sn - /-Lce)) is
asymptotically normal N(O, a 2 ce)). Under the null hypothesis the mean and variance are
equal to /-LCO) == 1/2 and a 2 CO) == 1/4, respectively, so that ,jnCSn - 1/2) NCO, 1/4).
The test that rejects the null hypothesis if ,jn CSn - 1/2) exceeds the critical value Za has
power function
TrnCe) = Po (.Jn(Sn - MC e )) > !Za - .Jn(MC e ) - MCO)))
( Iza - ,jn(FCO) - Fc-e)) )
== 1 - ep 2 + oCI).
ace)
Because FCO) - FC -e) > O for every e > O, it follows that for a == an ---* O sufficiently
slowly
Trn ce) --+ {
if e == O,
if e > O.
The limit power function corresponds to the perfect test with all error probabilities equal to
zero. D
The example exhibits a sequence of tests whose Cpointwise) limiting power function is
the perfect power function. This type of behavior is typical for all reasonable tests. The
point is that, with arbitrarily many observations, it should be possible to tell the null and
alternative hypotheses apart with complete accuracy. The power at every fixed alternative
should therefore converge to 1.
14.2 Definition. A sequence of tests with power functions e 1---+ Trn ce) is asymptotically
consistent at level ex at Cor against) the alternative e if it is asymptotically of level ex and
Trn ce) ---* 1. If a family of sequences of tests contains for every level ex E CO, 1) a sequence
that is consistent against every alternative, then the corresponding tests are simply called
consistent.
Consistency is an optimality criterion for tests, but because most sequences of tests are
consistent, it is too weak to be really useful. To make an informative comparison between
sequences of C consistent) tests, we shall study the performance of the tests in problems
that become harder as more observations become available. One way of making a testing
problem harder is to choose null and alternative hypotheses closer to each other. In this
section we fix the null hypothesis and consider the power at sequences of alternative s that
converge to the null hypothesis.
194 Relative Efficiency of Tests
Ha H 1
Figure 14.1. Asymptotic power function.
14.3 Example (Sign test, continued). Consider the power of the sign test at sequences of
alternative s en O. Suppose that the nuU hypothesis Ho : e == O is rejected if -JYi(Sn - ) >
Za. Extension of the argument of the preceding example yields
( 1. z - Ifi ( F ( O ) - F ( -e )) )
Jr n (en) == 1 - <p 2 a V fI, n + 0(1).
a(e n )
Since aCO) == , the levels Jrn(O) of the tests converge to <p(za) == a. The asymptotic
power at en depends on the rate at which en O. If en converges to zero fast enough to
ensure that -JYi ( F (O) - F ( -en)) O, then the power Jr n (en) converges to a: the sign test is
not able to discrirninate these alternative s from the nuU hypothesis. If en converges to zero
at a slow rate, then -JYi (F (O) - F ( -en)) 00, and the asymptotic power is equal to 1:
these a1ternatives are too easy. The intermediate rates, which yield a nontrivial asymptotic
power, appear to be of most interest. Suppose that the underlying distribution function F
is differentiable at zero with positive derivative f (O) > O. Then
-JYi(F(O) - F( -en)) == -JYi enf(O) + o(e n ).
This is bounded away from zero and infinity if en converges to zero at rate en == O (n- 1 / 2 ).
For such rates the power Jr n (en) is asymptoticalIy strictly between a and 1. In particular,
for every h,
Jr n ( :n ) -+ 1 - <t>(za - 2hf(O)).
The form of the limit power function is shown in Figure 14.1. O
In the preceding example only alternative s en that converge to the nun hypothesis at
rate O (1/ -JYi) lead to a nontrivial asymptotic power. This is typical for parameters that
depend "smoothly" on the underlying distribution. In this situation a reasonable method
for asymptotic comparison of two sequences of tests for Ho : e == O versus Ho : e > O is to
consider locallimiting power functions, defined as
Jr(h) == lim Jr n ( ) ,
n-+oo V n
h > O.
These limit s typicalIy exist and can be derived by the same method as in the preceding
example. A general scheme is as folIows.
Let e be a real parameter and let the tests reject the nulI hypothesis Ho : e == O for large
values of a test statistic Tn. Assume that the sequence Tn is asymptoticalIy normal in the
14.1 Asymptotic Power Functions
195
sense that, for all sequences of the form en == hl,Jn,
,Jn (Tn - p, (en) ) !!::. N (O, 1).
(J(e n )
(14.4)
Often JL(e) and (J2(e) can be taken to be the mean and the variance of Tn, but this is not
necessary. Because the convergence (14.4) is under a law indexed by en that changes with
n, the convergence is not implied by
,Jn (Tn - JL(e)) ()
cr(e) "'" N(O, 1),
every e .
( 14.5)
On the other hand, this latter convergence uniformly in the parameter e is more than is
needed in (14.4). The convergence (14.4) is sometimes referred to as "locally uniform"
asymptotic normality. "Contiguity arguments" can reduce the derivation of asymptotic
normality under en == hl,Jn to derivati on under e == O. (See section 14.1.1).
Assumption (14.4) includes that the sequence ,Jn(Tn - JL(O)) converges in distribution
to a normal N(O, (J2(0) )-distribution under e == O. Thus, the tests that reject the nulI
hypothesis Ha : e == O if ,Jn(Tn - JL(O)) exceeds (J(O)za are asymptotically oflevel a. The
power functions of these tests can be written
7T: n (e n ) = Pen (.fi!(T n -p,(e n ») > cr(O)za - .fi!(p,(e n ) -p,(0»)).
For en == hl,Jn, the sequence ,Jn (JL (en) - JL (O) ) converges to h JL' (O) if JL is differentiable
at zero. If (J(e n ) -+ (J(O), then under (14.4)
( h ) ( JL' (O) )
7T: n ,Jn 1 - <t> Za - h cr(O) .
(14.6)
For easy reference we formulate this result as a theorem.
14.7 Theorem. Let JL and (J befunctions of e such that (14.4) holdsfor every sequence
en == hl,Jn. Suppase that JL is differentiable and that (J is continuous at e == O. Then
the power functions Jr n of the tests that reject Ha : e == O for large values of Tn and are
asymptotically of level ex satisfy (14.6) for every h.
The limiting power function depends on the sequence of test statistics only through the
quantity JL' (O) I (J (O). This is called the slope of the sequence of tests. Two sequences
of tests can be asymptotically compared by just comparing the sizes of their slopes. The
bigger the slope, the better the test for Ha : e == O versus HI : e > O. The size of the slope
depends on the rate JL' (O) of change of the asymptotic mean of the test statistics relative to
their asymptotic dispersion (J (O). A good quantitative measure of comparison is the square
of the quotient of two slopes. This quantity is called the asymptotic relative efficiency and
is discussed in section 14.3.
If () is the only unknown parameter in the problem, then the available tests can be ranked
in asymptotic quality simply by the value of their slopes. In many problem s there are also
nuisance parameters (for instance the shape of a density), and the slope is a function of the
nuisance parameter rather than a number. This complicates the comparison considerably.
For every value of the nuisance parameter a different test may be best, and additional criteria
are needed to choose a particular test.
196
Relative Efficiency of Tests
14.8 Example (Sign test). According to Example 14.3, the sign test has slope 2 I (O). This
can also be obtained from the preceding theorem, in which we can choose {L(e) == 1- F ( -e)
and er 2 (e) == (1 - F ( - e) ) F ( - e) . O
14.9 Example (t-test). Let XI, . . . , X n be arandom sample from a distribution with mean
e and finite variance. The t -test rejects the null hypothesis for large values of . The sample
variance S2 converges in probability to the variance er 2 of asingle observation. The central
limit theorem and Slutsky's lemma give
r:: . ( X h/ ) (X - h/ ) ( 1 1 ) h/
",.;n - - == + h - - - 'v'-7 N(O,l).
S er S S a
Thus Theorem 14.7 applies with {L(e) == e la and a(e) == 1. The slope of the {-test equals
1/a. t D
14.10 Example (Sign versus t-test). Let XI, . . . , X n be arandom sample from a density
f(x - e), where I is symmetric about zero. We shall compare the performance of the
sign test and the (-test for testing the hypothesis Ha: e == O that the observations are
syrnmetrically distributed about zero. Assume that the distribution with density Ihas a
unique median and a finite second moment.
It suffices to compare the slopes of the two tests. By the preceding examples these
are 21(0) and (f x 2 I(x) dx) -1/2, respectively. Clearly the outcome of the comparison
depends on the shape I. It is interesting that the two slopes depend on the underlying shape
in an almost orthogonal manner. The slope of the sign test depends only on the height
of I at zero; the slope of the t -test depends mainly on the tails of f. For the standard
normal distribution the slopes are .J21n and 1. The superiority of the (-test in this case is
not surprising, because the t -test is uniformly most powerful for every n. For the Laplace
distribution, the ordering is reversed: The slopes are 1 and .J2. The superiority of the sign
test has much to do with the "unsmooth" character of the Laplace density at its mode.
The relative efficiency of the sign test versus the t - test is equal to
4f2(0) f x 2 f(x)dx.
Table 14.1 summarizes these numbers for a selection of shapes. For the uniform distribution,
the relative efficiency of the sign test with respect to the t -test equals 1/3. It can be shown
that this is the minimal possible value over all densities with mode zero (problem 14.7). On
the other hand, it is possible to construct distributions for which this relative efficiency is
arbitrarily large, by shifting mass into the tails of the distribution. The sign test is "robu st"
against heavy tails, the t-test is not. D
The simplicity of comparing slopes is attractive on the one hand, but indicates the
potential weakness of asymptotics on the other. For instance, the slope of the sign test was
seen to be I (O), but it is clear that this value alone cannot always give an accurate indication
t A1though (14.4) holds with this ehoice of 11- and a, it is not true that the sequence ,Jn ( x / S - e / a) is asymp-
totieally standard normal for every fixed e. Thus (14.5) is false for this ehoiee of 11- and a. For fixed e the
contribution of S - a to the limit distribution cannot be neglected, but for our present purpose it can.
14.1 Asymptotic Power Functions
197
Table 14.1. Relative efficiencies of
the sign test versus the t -test for
same distributions.
Distribution
Efficiency (signJt-test)
n 2 /12
2/n
2
1/3
Logistic
N ormal
Laplace
Uniform
of the quality of the sign test. Consider a density that is basically a normal density, but a tiny
proportion of 10- 1 °% of its total mass is located under an extremely thin but enormously
high peak at zero. The large value f (O) would strongly favor the sign test. Rowever, at
moderate sample sizes the observations would not differ significantly from a sample from
a normal distribution, so that the t -test is preferable. In this situation the asymptotics are
only valid for unrealistically large sample sizes.
Even though asymptotic approximations should always be interpreted with care, in the
present situation there is actually little to worry about. Even for n == 20, the comparison of
slopes of the sign test and the t -test gives the right message for the standard distributions
listed in Table 14.1.
14.11 Example (Mann- Whitney). Suppose we observe two independentrandom samples
XI, . .., X m and YI, ..., Y n from distributions F(x) and G(y - e), respectively. The base
distributions F and G are fixed, and it is desired to test the null hypothesis Ho : e == O
versus the alternative HI : e > O. Set N == m + n and assume that ml N ---+ A E (O, 1).
Furthermore, as sume that G has a bounded density g.
The Mann-Whitney test rejects the null hypothesis for large numbers of U ==
(mn)-1 Li Lj 1 {Xi < Yj}. By the two-sample U -statistic theorem
m
(U - Pe(X < Y)) == -- L:(G(X i - e) - EG(X i - e))
m . 1
1=
n
+ - L:(F(Y i ) - EF (Yi)) + oPe(l).
n . 1
J=
This readily yields the asymptotic normality (14.5) for every fixed e, with
lI(e) = 1 - f G(x - e) dF(x),
1 1
a 2 (e) == - var G(X - e) + varF(Y).
)... 1-)",
To obtain the local asymptotic power function, this must be extended to sequences eN ==
h I . It can be checked that the U -statistic theorem remains valid and that the Lindeberg
central limit theorem applies to the right side of the preceding display with eN replacing e.
Thus, we find that (14.4) holds with the same functions IL and a. (Alternatively we can use
contiguity and Le Cam' s third lemma.) Renee, the slope of the Mann-Whitney test equals
jJ/(O)la(O) == f g dF la (O). D
14.12 Example (Two-sample t-test). In the set-up of the preceding example suppose that
the base distributions F and G have equal means and finite variances. Then e == E(Y - X)
198
Relative Efficiency of Tests
Table 14.2. Relative efficiencies of the Mann- Whitney
test versus the two-sample t -test if f == g equals
a number of distributions.
Distribution
Efficiency
(Mann-Whitney/two-sample t-test)
Logistic
Normal
Laplace
Uniform
t3
ts
c(l - x 2 ) V O
n 2 /9
3/n
3/2
1
1.24
1.90
108/125
- -
and the t -test rejeets the null hypothesis Ho : () == O for large values of the statistie (Y - X) I S,
where S2 I N == S Im + S I n is the unbiased estimator of var( Y - X). The sequence S2
eonverges in probability to a 2 == varXI"A + varYI(l - "A). By Slutsky's lemma and the
eentrallimit theorem
0V C' x - h/-: ) hdf N(O, 1).
Thus (14.4) is satisfied and Theorem 14.7 applies with f.L(e) == ela and a(()) == 1. The
slope of the t-test equals f.L/(O)la(O) == Ila. O
14.13 Example (t-Test versus Mann- Whitney test). Suppose we observe two indepen-
dent random samples XI, . . . , X m and YI, . . . , Y n from distributions F (x) and G(x - ()),
respeetively. The base distributions F and G are fixed and are assumed to have equal means
and bounded densities. It is desired to test the null hypothesis Ho: e == O versus the
alternative HI : () > O. Set N == m + n and assume that ml N -+ "A E (O, 1).
The slopes of the Mann-Whitney test and the t-test depend on the nuisanee parameters
F and G. Aeeording to the preeeding examples the relative efficiency of the two sequences
of tests equals
((1- "A)var X +"A varY)(J gdF)2
(1 - "A) varo G(X) + "A varo F(Y)
In the important case that F == G, this expression simplifies. Then the variables G(X) and
F (Y) are uniformi y distributed on [O, 1]. Renee they have variance 1/ 12 and the relative
efficiency reduces to 12 var X(J f2(y) dy)2. Table 14.2 gives the relative efficiency if
F == G are both equal to a number of standard distributions. The Mann- Whitney test is
inferior to the t -test if F == G equals the normal distribution, but better for the logistic,
Laplace, and t -distribution. Even for the normal distribution the Mann- Whitney test does
remarkably well, with arelative efficiency of 3 /n 95%. The density that is proportional
to (1 - x 2 ) v O (and any member of its scale family) is least favorable for the Mann-
Whitney test. This density yields the lowest possible relative efficiency, which is stiH equal
to 108/125 86% (problem 14.8). On the other hand, the relative efficieney of the Mann-
Whitney test is large for heavy-tailed distributions the supremum value is infinite. Together
with the fact that the Mann- Whitney test is distribution-free under the nun hypothesis, this
14.2 Consistency
199
makes the Mann-Whitney test a strong competitor to the t-test, even in situations in which
the underlying distribution is thought to be approximately norma!. D
*14.1.1 Using Le Cam's Third Lemma
In the preceding examples the asymptotic normality of sequences of test statistics was
established by direct methods. For more complicated test statistics the validity of (14.4)
is easier checked by means of Le Cam's third lemma. This is illustrated by the following
example.
14.14 Example (Median test). In the two-sample set-up of Example 14.11, suppose that
F == G is a continuous distribution function with finite Fisher information for location 19.
The median test rejects the null hypothesis Ho : () == O for large values of the rank statistic
TN == N-I 'L::m+l1{ RNi < (N +1)j2}. By therankcentral limittheorem, Theorem 13.5,
under the null hypothesis,
( TN - ) = - f){F(Xi) < 1/2}
2N N N i=1
n
+ 2){F(Yj) < 1/2} +op(l).
N N j=1
Under the null hypothesis the sequence of variables on the right side is asymptotically
normal with mean zero and variance a 2 (0) == A(l - A)j4. By Theorem 7.2, for every
()N == hj-Jli,
TIif(Xi)TIjg(Yj-()N) h -J 1-A g' 1 2
log == - -(Yi) - -h (1 - A)I g + op(l).
TIi f(X i ) TI j g(Y j ) vn j=1 g 2
By the multivariate central limit theorem, the linear approximations on the right sides
of the two preceding displays are jointly asymptotically norma!. By Slutsky's lemma
the same is true for the left sides. Consequently, by Le Cam's third lemma the sequence
-Jli (T N - nj (2N)) converges under the alternative s ()N == h j -Jli in distribution to a
normal distribution with variance a 2 (0) and mean the asymptotic covariance T(h) of the
linear approximations. This is given by
1 f'
r(h) == -hA(l - A) -(y) dF(y).
F(y)::::1/2 f
Conclude that (14.4) is valid with f.-L «()) == T «()) and a «()) == a (O). (U se the test stati stics
T N - nj (2N) rather than T N .) The slope of the median test is given by - 2 y' A (1 - A) 10 1 / 2
(f' jf)(F- 1 (u)) du. D
14.2 Consistency
After noting that the power at fixed alternative s typically tends to 1, we focused attention
on the performance of tests at altematives converging to the null hypothesis. The compar-
ison of local power functions is only of interest if the sequences of tests are consistent at
200
Relative Efficiency of Tests
fixed alternatives. Fortunately, establishing consistency is rarely a problem. The following
lemmas describe two basic methods.
14.15 Lemma. Let Tn be a sequence ofstatistics such that Tn /-L(e) for every e. Then
the family of tests that reject the null hypothesis Ho : e == O for large values ofTn is consistent
against every e such that /-L(e) > /-L(O).
14.16 Lemma. Let /-L and (5 be functions of e such that (14.4) holds for every sequence
en == hl,Jn. Suppose that /-L is differentiable and that (5 žs contžnuous at zero, with /-L' (O) > O
and (5 (O) > O. Suppose that the tests that reject the null hypothesis for large values of Tn
possess nondecreasžng power functions e Trn (e). Then this family of tests is consistent
against every alternative e > O. Moreover, ijTr n (O) -+ ex, then Trn (en) -+ ex or Trn (en) -+ 1
when ,Jn en -+ O or ,Jn en -+ 00, respectively.
Proofs. For the first lemma, suppose that the tests reject the nun hypothesis if Tn exceeds
the critical value cn. By assumption, the probability under e == O that Tn is outside the
interval (/-L (O) - £, /-L (O) + £) converges to zero as n -+ 00, for every fixed E > O. If
the asymptotic level limPo(Tn > cn) is positive, then it follows that C n < /-L(O) + £
eventually. On the other hand, under e the probability that Tn is in (/-L (e) - £, /-L (e) + £ )
converges to 1. For sufficiently small £ and /-L (e) > /-L (O), this interval is to the right of
/-L(O) + £. Thus for sufficiently large n, the power Pe(Tn > cn) can be bounded below by
Pe (Tn E (/-L(e) - £, /-L(e) + £)) -+ 1.
For the proof of the second lemma, first note that by Theorem 14.7 the sequence of local
power functions Trn (hi ,Jn) converges to Tr(h) == 1 - <I>(Za - hJL'(O) 1(5 (O) ), for every h,
if the asymptotic level is ex. If,Jn en -+ O, then eventually en < hl,Jn for every given
h > O. By the monotonicity of the power functions, Trn (en) < Trn (hl,Jn) for sufficiently
large n. Thus lim sup Trn (en) < Tr(h) for every h > O. For h + O the right side converges
to Tr(O) == ex. Combination with the inequality Trn (en) > Trn(O) -+ a gives Trn (en) -+ ex.
The case that ,Jn en -+ 00 can be handled s imilarly. Finally, the power Trn (e) at fixed
alternative s is bounded below by Trn (en) eventually, for every sequence en + O. Thus
Trn (e) -+ 1, and the sequence of tests is consistent at e. .
The following examples show that the t-test and Mann-Whitney test are both consistent
against large sets of alternatives, albeit not exactly the same sets. They are both tests
to compare the locations of two samples, but the pertaining definitions of "location" are
not the same. The t -test can be considered a test to detect a difference in mean; the
Mann-Whitney test is designed to find a difference of P(X < Y) from its value 1/2 under
the null hypothesis. This evaluation is justified by the following examples and is further
underscored by the consideration of asymptotic efficiency in nonparametric model s . It is
shown in Section 25.6 that the tests are asymptotically efficient for testing the parameters
EY - EX or P(X < Y) if the underlying distributions F and G are completely unknown.
- -
14.17 Example (t-test). The two-sample t-statistic (Y - X)I S converges in probability to
E(Y - X) 1(5, where (52 == lim var(Y - X). If the null hypothesis postulates that EY == EX,
then the test that rejects the null hypothesis for large values of the t -statistic is consistent
against every alternative for which EY > EX. D
14.3 Asymptotic Relative Efficiency
201
14.18 Example (Mann- Whitney test). The Mann-Whitney statistic U converges in prob-
ability to P(X < Y), by the two-sample U -statistic theorem. The probability P(X < Y) is
equal to 1/2 if the two samples are equal in distribution and possess a continuous distribution
function. If the nulI hypothesis postulates that P(X < Y) == 1/2, then the test that rejects
for large values of U is consistent against any alternative for which P(X < Y) > 1/2. D
14.3 Asymptotic Relative Efficiency
Sequences of tests can be ranked in quality by comparing their asymptotic power functions.
For the test statistics we have considered so far, this comparison only involves the "slopes"
of the tests. The concept of relative efficiency yields a method to quantify the interpretation
of the slopes.
Consider a sequence of testing problems consisting of testing a nun hypothesis Ho : e == O
versus the alternative HI : e == (1)' We use the parameter v to describe the asymptotics;
thus v ---+ 00. We require a priori that our tests attain asymptotically level a and power
Y E (a, 1). Usually we can meet this requirement by choosing an appropriate number of
observations at "time" 1). A larger number of observations allows smaller level and higher
power. If TC n is the power function of a test if n observations are available, then we define
n 1) to be the minimal number of observations such that both
TC nu (O) < a, and TC nu (e1)) > Y.
If two sequences of tests are available, then we pre fer the sequence for which the numbers
n1) are smallest. Suppose that n1),1 and n1),2 observations are needed for two given sequences
of tests. Then, if it exists, the limit
1 . n1),2
lm-
1) ---HX) n 1), 1
is called the (asymptotic) relative efficiency or Pitman efficiency of the first with respect
to the second sequence of tests. Arelative efficiency larger than 1 indicates that fewer
observations are needed with the first sequence of tests, which may then be considered the
better one.
In principle, the relative efficiency may depend on a, y and the sequence of alternative s
(1). The concept is mostly of interest if the relative efficiency is the same for all possible
choices of these parameters. This is often the case. In particular, in the situations considered
previously, the relative efficiency turns out to be the square of the quotient of the slopes.
14.19 Theorem. Consider statistical models (Pn,e: e > O) such that II Pn,e - Pn,o II ---+ O
as e ---+ O, for every n. Let Tn,1 and Tn,2 be sequences of statistics that satisfy (14.4) for
every sequence en {. O and functions ILi and (Ji such that ILi is difJerentiable at zero and (Ji
is continuous at zero, with IL (O) > O and (Ji (O) > O. Then the relative efficiency of the
tests that reject the nuli hypothesis Ho : e == O for large values of Tn,i is equal to
( IL (O)/(JI (O) ) 2 ,
IL; (O) / (J2 (O)
for every sequence of alternatives (1) {. O, independently of a > O and Y E (a, 1). If the
powerfunctions of the tests based on Tn,i are nondecreasingforevery n, then the assumption
202
Relative Efficiency of Tests
of asymptotic normality ofTn,i can be relaxed to asymptotic normality under every sequence
en == 0(1/ ) only.
Proof. Fix ex and y as in the introduction and, given alternative s ()v O, let nv,i observations
be used with each of the two tests. The assumption that II Pn,Đ v - Pn,a II -+ O as v -+ 00 for
each fixed n forces nv,i -+ 00. Indeed, the sum of the probabilities of the first and second
kind of the test with critical region Kn equals
r d Pn,O + r d P n ,()" = 1 + r (Pn,O - Pn,(}") d{.tn.
JK n JK JK n
This sum is minimized for the critical region Kn == {Pn,a - Pn,e v < O}, and then equals
1 - II Pn,Đ v - Pn,a II. By assumption, this converges to 1 as v -+ 00 uniformly in every
finite set of n. Thus, for every bounded sequence n == nv and any sequence of tests, the
sum of the error probabilities is asymptotically bounded below by 1 and cannot be bounded
above by ex + 1 - y < 1, as required.
Now that we have ascertained that nv,i -+ 00 as v -+ 00, we can use the asymptotic
norrnality of the test statistics Tn, i. The convergence to a continuous distribution implies that
the asymptotic level and power attained for the minimal numbers of observations (minimal
for obtaining at most level ex and at least power y) is exactly a and y. In order to obtain
asymptotic level ex the tests must reject Ha if (Tnv,i - J1-i (O)) > ai (O)Za + 0(1). The
powers of these tests are equal to
( J1- (O) ( ))
Jrn",i(e V ) = 1- <I> Za +0(1) - y'rl;ie v (T;(O) 1 +0(1) +0(1).
This sequence of powers tends to y < 1 if and only if the argument of ep tends to Zy. Thus
the relative efficiency of the two sequences of tests equals
lim n v ,2 == lim n v ,2 e ?; == (za - Zy)2 / (Za - Zy)2 .
v oo n v , I v oo nv, I eJ (J1- (O) / a2 (O) ) 2 (J1- (O) / (JI (O) ) 2
This proves the first assertion of the theorem.
If the power functions of the tests are monotone and the test statistics are asymptotically
normal for every sequence en == O (1/ ), then 7Tn,i (en) -+ a or 1 if en -+ O or 00,
respectivel y (see Lernma 14.16). In that case the sequences of tests can only meet the (a, y)
requirement for testing altematives e v such that e v == O (1). For such sequences the
preceding argument is valid and gives the asserted relative efficiency. .
*14.4 Other Relative Efficiencies
The asymptotic relative efficiency defined in the preceding section is known as the Pitlnan
relative efficiency. In this section we discuss same other types of relative efficiencies. Define
ni (ex, y, e) as the minimal numbers of observations needed, with i E {I, 2} for two given
sequences of tests, to test a null hypothesis Ha : e == O versus the alternative HI : e == e
at level ex and with power at least y. Then the Pitman efficiency against a sequence of
alternatives e v -+ O is defined as (if the limits exists)
. n2(ex, y, e v )
lIm .
v oo nI (a, y, e v )
14.4 Other Relative Efficiencies
203
The devi ce to let the alternative s e v tend to the null hypothesis was introduced to make the
testing problem s harder and harder, so that the required numbers of observations tend to
infinity, and the comparison becomes an asymptotic one. There are other possibilities that
can serve the same end. The testing problem is harder as a is smaller, as y is 1 arg er, and
(typically) as e is c10ser to the nulI hypothesis. Thus, we could also let a tend to zero, or y
tend to one, keeping the other parameters fixed, or even let two or all three of the parameters
vary. For each possib1e method we cou1d define the relative efficiency of two sequences of
tests as the limit of the quotient of the minima1 numbers of observations that are needed.
Most of these possibi1ities have been studied in the literature. N ext to the Pitman efficiency
the most popu1ar efficiency measure appears to be the Bahadur efficiency, which is defined
as
1 . n2(av,y,e)
Im .
v oo nI (a v , y, e)
Here a v tends to zero, but y and e are fixed. Typically, the Bahadur efficiency depends on
e, but not on y, and not on the particular sequence a v + O that is used.
Whereas the ca1cu1ation of Pitman efficiencies is most often based on distributiona11imit
theorems, Bahadur efficiencies are derived from large deviations results. The reason is that
the probabilities of first ar second kind for testing a fixed null hypothesis against a fixed
alternative usualIy tend to zero at an exponential speed. Large deviations theorems quantify
this speed. Suppose that the null hypothesis Ho : e == O is rejected for large va1ues of a test
statistic Tn, and that
2
--logPo(Tn > t) e(t),
n
every t,
(14.20)
Po
Tn JJ-(e).
(14.21)
The first result is a large deviation type result, and the second a "law of large numbers."
The observed significance level of the test is defined as Po(Tn > t)lt=T n . Under the nulI
hypothesis, this random variab1e is uniformly distributed if Tn possesses a continuous
distribution function. For a fixed alternative e, it typicalIy converges to zero at an exponential
rate. For instance,-under the preceding conditions, if e is continuous at JJ-(e), then (because
e is necessari1y monotone) it is immediate that
2 Po ( )
- -log Po(Tn > t)lt=T n e JJ-(e) .
n
The quantity e(JJ-(e)) is called the Bahadur slope of the test (orratherthe limit in probability
of the 1eft side if it exists). The quotient of the slopes of two sequences of test statistics
gives the Bahadur re lative efficiency.
14.22 Theorem. Let Tn,l and Tn,2 be sequences of statistics in statistical models (Pn,O,
Pn,e) that satisfy (14.20) and (14.21) for functions e ž and numbers JJ-ž(e) such that e ž is
continuous at JJ-ž (e). Then the Bahadur relative efficiency of the sequences of tests that
reject for large values of Tn,ž is equal to el (JJ-I (e)) / e2 (JJ-2 (e)), for every a v + O and every
1 > y > sUPn Pn,e (Pn,O == O).
Proof. For simplicity of notation, we drop the index i E {1, 2} and write n v for the minima1
numbers of observations needed to obtain 1evel a v and power y with the test statistics Tn.
204
Relative Efficiency of Tests
The sample sizes nv necessarily converge to 00 as 1) ---+ 00. Ifnot, then there would exist
a fixed value n and a (sub )sequence of tests with levels tending to O and powers at least y .
However, for any fixed n, and any sequence of measurable sets Km with Pn,o(K m ) --+ O as
m ---+ 00, the probabilities P n ,8 (Km) == P n ,8 (Km n Pn,O == O) + 0(1) are eventually strictly
smalIer than y, by assumption.
The most powerful level C¥v-test that rejects for large values of Tn has critical re-
gion {Tn > cn} or {Tn > cn} for C n == inf{ C : Po(Tn > c) < c¥v}, where we use > if
P O C01 > cn) < C¥v and> otherwise. Equivalently, with the notation Ln == Po(Tn > t)lt=T n ,
this is the test with critical region {Ln < c¥v}. By the definition of nv we conclude that
( 2 2 ){ >y
P n ,8 - - log Ln > - - log C¥v -
n n < y
for n == n v ,
for n == nv - 1.
By (14.20) and (14.21), the random variable inside the probability converges in probability
to the number e(/.-L(e)) as n ---+ 00. Thus, the probability converges to O or 1 if -(2j n) log C¥v
is asymptotically strictly bigger or smaller than e(/.-L(e)), respectively. Conclude that
2
lim sup - -log C¥v < e(/.-L(e))
v --HX) n v
2
lim inf - log C¥v > e (/.-L(e)).
v oo nv - 1
Combined, this yields the asymptotic equivalence nv rv -2log c¥v/e(/.-L(e)). Applying this
for both n v ,l and n V ,2 and taking the quotient, we obtain the theorem. .
Bahadur and Pitman efficiencies do not always yield the same ordering of sequences
of tests. In numerical comparisons, the Pitman efficiencies appear to be more relevant for
moderate sample sizes. This is explained by their method of calculation. By the preceding
theorem, Bahadur efficiencies folIow from a large deviations result under the nulI hypothesis
and a law of large numbers under the alternative. A law of large numbers is of less accuracy
than a di s tributional limit result. Furthermore, large deviation results, while mathematicalIy
interesting, often yield poor approximations for the probabilities of interest. For instance,
condition (14.20) shows that Po(Tn > t) == exp(- ne(t)) expo(n). Nothing guarantees
that the term exp o(n) is close to 1.
On the other hand, often the Bahadur efficiencies as a function of e are more informa-
tive than Pitman efficiencies. The Pitman slopes are obtained under the condition that the
sequence -Jfi(Tn - /.-L(O)) is asymptoticalIy normal with mean zero and variance a 2 (0).
Suppose, for the present argument, that Tn is normally distributed for every finite n, with
the parameters /.-L(O) and CT 2 (0)jn. Then, because 1 - <p(t) rv cjJ(t)jt as t ---+ 00,
2 2 ( ( t -Jfi )) t2
--logPo(Tn > /.-L(O)+t)==--log 1-<p - ---+ 2 '
n n CT (O) CT (O)
The Bahadur slope would be equal to (/.-L(e) - /.-L(0))2 jCT 2 (0). For e ---+ O, this is approx-
imately equal to e 2 times the square of the Pitman slope /.-L' (0)2 / CT 2 (O). Consequently, the
limit of the Bahadur efficiencies as e ---+ O would yield the Pitman efficiency.
Now, the preceding argument is completely false if Tn is only approximately normalIy
distributed: Departures from normality that are negligible in the sense of weak convergence
need not be so for large-deviation probabilities. The difference between the "approximate
every t.
14.4 Other Relative Efficiencies
205
Bahadur slopes" just obtained and the true slopes is often substantial. However, the argument
tends to be "more correct" as t approaches /-L(O), and the conclusion that limiting Bahadur
efficiencies are equal to Pitman efficiencies is often correct. t
The main tool needed to evaluate Bahadur efficiencies is the large-deviation result
(14.20). For averages Tn, this follows from the Cramer-Chernoff theorem, which can
be thought of as the analogue of the central limit theorem for large deviations. It is a refine-
ment of the weak law of large numbers that yields exponential convergence of probabilities
of deviations from the mean.
The cumulant generating function of arandom variable Y is the function u f--* K (u) ==
log Ee uY . If we allow the value 00, then this is well-defined for every u E JR. The set of u
such that K (u) is finite is an interval that may or may not contain its boundary points and
may be just the point {O}.
14.23 Proposition (Cramer-Chernofftheorem). Let YI, Y 2 , . .. he i.i.d. random vari-
ahles with cumulant generating function K. Then, for every t,
1 -
-logP(Y > t) inf(K(u) - tu).
n u o
Proof. The cumulant generating function of the variables Yi - t is equal to u f--* K (u) - ut.
Therefore, we can restrict ourselves to the case t == O. The proof consists of separate upper
and lower bounds on the probabilities P( Y > O).
The upper bound is easy and is valid for every n. By Markov's inequality, for every
u > 0,
- -
P(Y > O) == P(e unYn > 1) < Ee unYn == enK(u).
Take logarithms, divide by n, and take the infimum over u > O to find one half of the
proposition.
For the proof of the lower bound, first consider the cases that Yi is nonnegative or
nonpositive. If P(Y i < O) == O, then the function u f--* K (u) is monotonely increasing on JR
and its infimum on u > O is equal to O (attained at u == O); this is equal to n-I log P(Y > O)
for every n. Second, if P(Y i > O) == O, then the function u f--* K(u) is monotonely
decreasing on JR with K(oo) == 10gP(Y I == O); this is equal to n-IlogP(Y > O) for every
n. Thus, the theorem is valid in both cases, and we may exclude them from now on.
First, assume that K (u) is finite for every u E JR. Then the function u f--* K (u) is analytic
on JR, and, by differentiating under the expectation, we see that K ' (O) == EY I . Because Yi
takes both negative and positive values, K(u) 00 as u :1:00. Thus, the infimum of
the function u f--* K (u) over u E JR is attained at a point Uo such that K ' (uo) == O.
The case that Uo < O is trivial, but requires an argument. By the convexity of the function
u f--* K (u), K is nondecreasing on [uo, (0). If Uo < O, then it attains its minimum value over
u > O at u == O, which is K (O) == O. Furthermore, in this case EY! == K ' (O) > K ' (uo) == O
(strict inequality under our restrictions, for instance because K " (O) == var YI > O) and
hence P(Y > O) 1 by the law of large numbers. Thus, the limit of the left side of the
proposition (with t == O) is O as well.
t In [85] a preeise argument is given.
206
Relative Efficiency of Tests
For Uo > O, let ZI, Z2, . . . be i.i.d. random variables with the distribution given by
dPz(z) == e-K(u o ) e Uoz dPy(z).
Then ZI has cumulant generating function u 1---+ K(uo + u) - K(uo), and, as before, its
mean can be found by differentiating this function at u == O: EZ I == K' (uo) == O. For every
E > O,
-
P(Y > O) == El {Zn > O}e-uonZn enK(u o )
> P(O < Zn < E) e- uons enK(u o ).
- -
Because Zn has mean O, the sequence P(O < Zn < 8) is bounded away from O, by the
central limit theorem. Conclude that n -1 times the limit inferior of the logarithm of the
left side is bounded below by -UOE + K(uo). This is true for every 8 > O and hence also
for E == O.
Finally, we remove the restriction that K (u) is finite for every u, by a truncation argument.
For a fixed, large M, let Y f1, y 2 M , . . . be distributed as the variables YI, Y 2 , . . . given that
I Yi I < M for every i, that is, they are i.i.d. according to the conditional distribution of YI
given IYII < M. Then, with u 1---+ KM (u) == logEe uY1 1{IY I I < M},
liminf logP(Y > O) > log(P(f: > O)p(1 y i M I < Mr)
> inf KM (u),
u o
by the preceding argument applied to the truncated variables. Let s be the limit of the right
side as M --+ 00, and let AM be the set {u > O: KM (u) < s}. Then the sets AM are
nonempty and compact for sufficiently large M (as soan as KM (u) --+ 00 as u --+ ::i: (0),
with Al ::J A 2 ::J . . ., whence nAM is nonempty as well. Because KM converges pointwise
to K as M --+ 00, any point Ul E nAM satisfies K(UI) == limKM(uI) < s. Conclude that
s is bigger than the right side of the proposition (with t == O). .
14.24 Example (Sign statistic). The cumulant generating function of a variable Y that
is -1 and 1, each with probability !, is equal to K (u) == log cosh u. Its derivative is
K' (u) == tanh u and hence the infimum of K (u) - t u over u E JR is attained for u == arctanh t.
By the Cramer-Chernoff theorem, for O < t < 1,
2 -
- -log P(Y > t) --+ e(t) : == -210g cosh arctanh t + 2t arctanh t.
n
We can apply this result to find the Bahadur slope of the sign statistic Tn == n-I L:7=I sign(X i ).
If the nuU distribution of the random variables XI, . . . , X n is continuous and symmetric
about zero, then (14.20) is valid with e(t) as in the preceding display and with /.L(e) ==
Ee sign(X I ). Figure 14.2 shows the slopes of the sign statistic and the sample mean for
testing the location of the Laplace distribution. The local optimality of the sign statistic is
reflected in the Bahadur slopes, but for detecting large differences of location the mean is
better than the sign statistic. However, it should be noted that the power of the sign test in
this range is so close to 1 that improvement may be irrelevant; for example, the power is
0.999 at level 0.007 for n == 25 at e == 2. O
14.4 Other Relative Efficžencies
207
o
C\J
...-
...-
LO
o
o
o
O 1 2 3 4
Figure 14.2. Bahadur slopes of the sign statistic (solid line) and the sample mean (dotted line) for
testing that arandom sample from the Laplace distribution has mean zero versus the alternative that
the mean is e, as a function of e.
14.25 Example (Student statistic). Suppose that XI, . . . , X n are arandom sample from a
normal distribution with mean Ih and variance a 2 . We shall consider a known and compare
the slopes of the sample mean and the Student statistic X ni Sn for testing Ha : f.L == O.
The cumulant generating function of the normal distribution is equal to K (u) == u IL +
u2a2. By the Cramer-Chernofftheorem, for t > O,
2 _ t 2
--logPa(Xn > t) e(t): == 2'
n a
Thus, the Bahadur slope of the sample mean is equal to Ih 2 I a 2 , for every Ih > O.
Under the null hypothesis, the statistic -JnXnl Sn possesses the t-distribution with (n -1)
degrees of freedom. Thus, for arandom sample Za, ZI, . . . of standard normal variables,
for every t > O,
p ( {li X n > t ) == P ( tLI > t 2 ) == p ( Z2 _ t2 Z2 > O ) .
a V;=I Sn - 2 n-I - 2 a 8 l -
This probability is not of the same form as in the Cramer-Chernoff theorem, but it concerns
almost an average, and we can obtain the large deviation probabilities from the cumulant
generating function in an analogous way. The cumulant generating function of a square
of a standard normal variable is equal to u 1---+ - log(l - 2u), and hence the cumulant
generating function of the variable Z5 - t 2 'L7 11 Zf is equal to
Kn(u) == - log(l - 2u) - (n - 1) log(l + 2t 2 u).
This function is nicely differentiable and, by straightforward calculus, its minimum value
can be found to be
1 ( t 2 + 1 ) 1 ( (n - 1)(t 2 + 1) )
inf Kn (u) == - - log 2 - - (n - 1) log.
u 2 t n 2 n
The minimum is achieved on [0,(0) for t 2 > (n - 1)-1. This expression divi ded by n is the
analogue of infu K (u) in the Cramer-Chemoff theorem. By an extension of this theorem,
208
Relative Efficiency of Tests
for every t > O,
_ 2 log Po ( (1l X n > t ) -+ e(t) == log(t 2 + 1).
n V Sn
Thus, the Bahadur slope of the Student statistic is equal to log(l + JL21(52).
For JL 1 (5 close to zero, the Bahadur slopes of the sample mean and the Student statistic
are close, but for large JL 1 (5 the slope of the sample mean is much bigger. This suggests
that the loss in efficiency incurred by unnecessarily estimating the standard deviation (5 can
be substantial. This suggestion appears to be unrealistic and also contradicts the fact that
the Pitman efficiencies of the two sequences of statistics are equal. D
14.26 Example (Neyman-Pearson statistic). The sequence of Neyman-Pearson statis-
tics I17=I (p()1 p()o) (Xi) has Bahadur slope -2P() log (Pe o 1 p()). This is twice the Kullback-
Leibler divergence of the measures p()o and P() and shows an important connection between
large deviations and the Kullback - Leibler divergence.
In regular cases this resu1t is a consequence of the Cramer -Chernoff theorem. The
variable Y == log P() 1 p()o has cumulant generating function K (u) == log f Pe pJo-u d JL un-
der p()o' The function K (u) is finite for O < u < 1, and, at least by formal calculus,
K' (1) == P() log(p() 1 p()o) == f.L(e), where JL(e) is the asymptotic mean of the sequence
n-I I: log(p()/p()o) (Xi)' Thus the infimum of the function u K(u) - Uf.L(e) is attained
at u == 1 and the Bahadur slope is given by
e(f.L(e)) == -2(K(1) - JL(e)) == 2P() log .
p()o
In section 16.6 we obtain this result by a direct, and rigorous, argument. D
For statistics that are not means, the Cramer-Chernoff theorem is not applicable, and we
ne ed other methods to compute the Bahadur efficiencies. An important approach applies
to functions of means and is based on more general versions of Cramer' s theorem. A first
generalization asserts that, for certain sets B, not necessarily of the form [t, 00),
1 -
-log P(Y E B) -+ -infyEBI (y),
n
I (y) == sup(uy - K (u)).
u
- -
For a given statistic of the form ep (Y), the large deviation probabilities of interest P( ep (Y) >
t) can be written in the form P(Y E Bt) for the inverse images Bt == ep-I[t, 00). If Bt is an
eligible set in the preceding display, then the desired large deviations result follows, although
we shalI still have to evaluate the repeated "inf sup" on the right side. N ow, according to
Cramer's theorem, the display is valid for every set such that the right side does not change
if B is replaced by its interior or its closure. In particular, if ep is continuous, then Bt is
closed and its interior Bt contains the set ep-I (t, 00). Then we obtain a large deviations
result if the difference set ep -1 {t} is "small" in that it does not play a role when evaluating
the right side of the display.
- -
Transforming a univariate mean Y into a statistic ep (Y) can be of interest (for example,
to study the two-sided test statistics I YI), but the real promise of this approach is in its
applications to multivariate and infinite-dimensional means. Cramer's theorem has been
generalized to these situations. Generallarge deviation theorems can best be formulated
14.4 Other Relative Efficiencies
209
as separate upper and lower bounds. A sequence of random map s X n : s1 1--+ JI]) from a
probability space (S1, U, P) into a topological space JI]) is said to satisfy the large deviation
principle with rate function I if, for every closed set F and for every open set G,
1
lim sup -logP*(Xn E F) < - inf I(y),
n-+oo n yEF
1
liminf -logP*(X n E G) > - inf I(y).
n-+oo n yEG
The rate function I : JI]) 1--+ [O, 00] is assumed to be lower semicontinuous and is called a
good rate function if the sublevel sets {y : I (y) < M} are compact, for every M E JR. The
inner and outer probabilities that X n belongs to a general set B is sandwiched between the
probabilities that it belongs to the interior B and the closure B . Thus, we obtain a large
deviation result with equality for every set B such that inf { I (y) : y E B } == inf { I (y) :
y E B}. An implication for the slopes of test statistics of the form ep (X n ) is as follows.
14.27 Lemma. Suppose that ep : JI]) 1--+ JR is continuous at every y such that I (y) < 00
and suppose that inf { I (y) : ep (y) > t} == inf { I (y) : ep (y) > t}. If the sequence X n satisfies
the large-deviation principle with the rate function I under Po, then Tn == ep (X n ) satisfies
(14.20) with e(t) == 2inf{I(y): ep(y) > tJ. Furthermore, if I is a good ratefunction, then
e is continuous at t.
Proof. Define sets At == ep-I (t, (0) and Bt == ep-I [t, (0), and let JI])o be the set where I is
finite. By the continuity of ep, B t n JI])o == Bt n JI])o and Bt n JI])o =:) At n JI])o. (If y Bt, then
there is anet yn E B with Yn y; if also y E JI])o, then ep(y) == lim ep (Yn) < t and hence
o
y At.) Consequently, the infimum of I over Bt is at least the infimum over At, which
is the infimum over Bt by assumption, and also the infimum over Bt. Condition (14.20)
follows upon applying the large deviation principle to Bt and B t.
The function e is nondecreasing. The condition on the pair (I, ep) is exactly that e is
right-continuous, because e(t+) == inf{ I (y) : ep (y) > t}. To prove the left-continuity
of e, let tm t t. Then e(t m ) t a for some a < e(t). If a == 00, then e(t) == 00 and
e is left-continuous. If a < 00, then there exists a sequence ym with ep (Ym) > tm and
21 (Ym) < a + 1/ m. By the goodness of I, this sequence has a converging subnet Ym' y.
Then 21 (y) < lim inf 21 (Ym') < a by the lower semicontinuity of I, and ep (y) > t by the
continuity of ep. Thus e(t) < 21 (y) < a. .
Empirical distributions can be viewed as means (of Dirac measures ), and are therefore
potential candidates for a large-deviation theorem. Cramer' s theorem for empirical di s tri -
butions is known as Sanov' s theorem. Let lLI (X, A) be the set of all probability measures
on the measurable space (X, A), which we assume to be a complete, separable metric space
with its Borel (5 -field. The T -topology on lLI (X, A) is defined as the weak topology gen-
erated by the collection of all maps P 1--+ P f for f ranging over the set of all bounded,
measurable functions on f : X 1--+ JR. t
14.28 Theorem (Sanov's theorem). Let P n be the empirical measure of a randOn'l sample
of size n from a fixed measure P. Then the sequence P n viewed as maps into lLI (X, A)
t For a pro of of the following theorem, see [31], [32], or [65].
210
Relative Efficiency of Tests
satisfies the large deviation principle relative to the '[ -topology, with the good rate Junction
I(Q) == -Qlogp/q.
For X equal to the real line, LI (X, A) can be identified with the set of cumulative
distribution functions. The '[ -topology is stronger than the topology obtained from the
uniform norm on the distribution functions. This follows from the fact that if both Fn (x) ---+
F(x) and Fn{x} ---+ F{x} for every x E JR, then IIFn - Flloo ---+ O. (see problem 19.9).
Thus any function ep that is continuous with respect to the uniform norm is also continuous
with respect to the i -topology, and we obtain a large collection of functions to which we
can apply the preceding lemrna. Trimmed means are just one example.
14.29 Example (Trimmed means). Let IF n be the empirical distribution function of a ran-
dom sample of size n from the distribution function F, and let JF 1 be the corresponding
quantile function. The function ep (IF n) == (1 - 2a)-1 J:- a F l (s) ds yields a version of the
a-trimmed mean (see Chapter 22). We assume that O < a < and (partly for simplicity)
that the nulI distribution Fo is continuous.
If we show that the conditions of Lemma 14.27 are fulfilled, then we can conclude, by
Sanov's theorem,
2 ( ) . Jo
- -log PFo ep (F n) > t ---+ e(t) :== 2 Inf -G log -.
n G: (G) t g
Because F n F uniform1y by the Glivenko-Cantelli theorem, Theorem 19.1, and epz is
continuous, ep (JF n) ep (F), and the Bahadur slope of the a-trimmed mean at an alternative
F is equal to e (<tJ (F) ) .
Finally, we show that ep is continuous with respect to the unifonn topology and that the
function t f-+ inf{ -G log(Jo/ g» : ep(G) > t} is right-continuous at t if Fo is continuous
at t. The map ep is even continuous with respect to the weak topology on the set of distri-
bution functions: If a sequence of measures G m converges weakly to a measure G, then
the corresponding quantile functions G;;/ converge weakly to the quantile function G- 1
(see Lemma 21.2) and hence ep(G m ) ---+ <tJ(G) by the dominated convergence theorem.
The function t f-+ inf{ -G log(Jo/ g) : ep(G) > t} is right-continuous at t if for every
G with <tJ(G) == t there exists a sequence G m with ep(G m ) > t and G m log(fo/ gm) ---+
G log(Jo/ g). If G log(Jo/ g) == -00, then this is easy, for we can choose any fixed G m that
is singular with respect to Fo and has a trimmed mean bigger than t. Thus, we may assume
that G Ilog(Jo/ g) I < 00, that G « Fo and hence that G is continuous. Then there exists a
point c such that a < G(c) < 1 - a. Define
dG m { 1_l
dG (x) == 1 + n
if x < c,
if x > c.
Then G m is a probability distribution for suitably chosen Cm > O, and, by the dominated
convergence G m log(Jo/ gm) ---+ G log(fo/ g) as m ---+ 00. Because G m (x) < G(x) for all
x, with strict inequality (at least) for all x < c such that G(x) > O, we have that G l (s) >
G- 1 (s) for all s, with strictinequality for all s E (O, G(c)]. HencethetrimmedmeancjJ(G m )
is strictly bigger than the trimmed mean cjJ(G), for every m. O
14.5 Rescaling Rates
211
*14.5 Rescaling Rates
The asymptotic power functions considered earlier in this chapter are the limits of "local
power functions" of the form h r-+ JT: n (h / -Jfi). The rescaling rate -Jfi is typical for testing
smooth parameters of the model. In this section we have a closer look at the rescaling rate
and discuss some nonregular situations.
Suppose that in a given sequence of models (X n , An, P n ,() : e E 8) it is desired to test the
null hypothesis Ho : e == e o versus the alternative s HI : e == en. For probability measures P
and Q define the total variation distance II P - Q" as the LI-distance f Ip - q I dJ-l between
two densities of P and Q.
14.30 Lemma. The power function JT: n of any test in (X n , An, P n ,() : e E 8) satisfies
JT: n (e) - JT: n (e o ) < \I P n ,() - Pn,()o II.
For any e and e o there exists a test whose power function attains equality.
Proof. If 1T n is the power function of the test 1n, then the difference on the left side can
be written as f 1n (Pn,() - Pn,()o) d J-ln' This expression is maximized for the test function
CPn == 1 {Pn,() > Pn,()o}' Next, for any pair ofprobability densities p and q we have fq>p (q -
p) dJL == f Ip - ql dJ-l, since f(p - q) dJ-l == O. .
This lemma implies that for any sequence of alternative s en:
(i) If II Pn,()n - Pn,()o II --+ 2, then there exists a sequence of tests with power 1Tn (en)
tending to 1 and size JT: n (e o ) tending to O (a perfect sequence of tests).
(ii) If II Pn,e n - Pn,()o II --+ O, then the power of any sequence of tests is asymptotically
less than the level (every sequence of tests is worthless).
(iii) If" Pn,()n - Pn,()o \I is bounded away from O and 2, then there exists no perfect sequence
of tests, but not every test is worthless.
The rescaling rate h / -Jfi used earlier sections corresponds to the third possibility. These ex-
amples concem model s with independent observations. Because the total variation distance
between product measures cannot be easily expressed in the distances for the individual
factors, we translate the results into the Hellinger distance and next study the implications
for product experiments.
The Hellinger distance H (P, Q) between two probability measures is the L 2 -distance
between the square roots of the corresponding densities. Thus, its square H 2 (P, Q) is equal
to f (,JP - -Jq)2 d J-l. The distance is convenient if considering product measures. First,
the Hel1inger distance can be expressed in the Hellinger affinity A(P, Q) == f -JP-Jq dJ-l,
through the formula
H 2 (P, Q) == 2 - 2A(P, Q).
N ext, by Fubini' s theorem, the affinity of two product measures is the product of the
affinities. Thus we arrive at the formula
H 2 (p n , Qn) == 2 _ 2(1 _ H 2 (P, Q))n.
212
Relative Efficiency of Tests
14.31 Lemma. Given a statistical model (Pe : e > e o ) set Pn,e == P;. Then the possi-
bilities (i), (ii), and (iii) arise when nH2(Pe n , Peo) converges to 00, converges to O, or is
bounded away from O and 00, respectively. In particular, if H2(Pe, Peo) == O (le - e O la)
as e ---+ e o , then the possibilities (i), (ii), and (iii) are valid when n l/a len - e o I converges to
00, converges to O, or is bounded away from O and 00, respectively.
Proof. The possibilities (i), (ii), and (iii) can equivalently be described by replacing the
total variation distance II P - P II by the squared Hellinger distance H2(P , P ). This
follows from the inequalities, for any probability measures P and Q,
H 2 (P, Q) < II P - QII < (2 - A 2 (P, Q)) 1\ 2H(P, Q).
The inequality on the left is immediate from the inequality I,JP - ,Jq12 < Ip - ql,
valid for any nonnegative numbers p and q. For the inequality on the right, first note
that pq == (p v q) (p 1\ q) < (p + q) (p 1\ q), whence A 2 (P, Q) < 2 f (p 1\ q) d JL, by the
Cauchy-Schwarz inequality. Now f (p 1\ q) d JL is equal to 1 - II p - Q II, as can be seen
by splitting the domains of both integrals in the sets p < q and p > q. This shows that
II p - Q II < 2 - A 2 (P, Q). That II P - Q II < 2H (P, Q) is a direct consequence of the
Cauchy-Schwarz inequality.
We now express the Hellinger distance of the product measures in the Hellinger distance
of Pen and Pe o and manipulate the nth power function to conclude the proof. .
14.32 Example (Smooth models). If the model (X, A, Pe : e E 8) is differentiable in
quadratic mean at e o , then H2(Pe, Peo) == O(le - 8 0 1 2 ). The intermediate rate of conver-
gence (case (iii)) is,Jn. D
14.33 Example (Uniform law). If Pe is the uniformmeasure on [O, eJ, then H2(Pe, Peo) ==
O ( I e - e o I). The intermediate rate of convergence is n. In this case we would study
asymptotic power functions defined as the limits of the local power functions of the form
h r-+ Jr n (e o + h / n). For instance, the level ex tests that reject the null hypothesis Ho : e == e o
for large values of the maximum X (n) of the observations have power functions
Jr n (80 + : ) = Plio+hjn(X(n) > 80(1 - a)l j n) 1 - (1 - a)e- hjlio .
Relative to this rescaling rate, the level ex tests that reject the null hypothesis for large values
of the mean X n have asymptotic power function ex (no power). D
14.34 Example (Triangular law). Let Pe be the probability distribution with density
x r-+ (1 - Ix - 81)+ on the real line. Some clever integrations show that H2(Pe, Po) ==
e2Iog(1/e) + 0(e 2 ) as e ---+ O. (It appears easiest to compute the affinity first.) This
leads to the intermediate rate of convergence ,J n log n. D
The preceding lemmas concern testing a given simple null hypothesis against a simple
alternative hypothesis. In many cases the rate obtained from considering simple hypotheses
does not depend on the hypotheses and is also globally attainable at every parameter in
the parameter space. If not, then the global problems have to be taken into account from
the beginning. One possibility is discussed within the context of density estimation in
section 24.3.
Problems
213
Lernma 14.31 gives rescaling rates for problems with independent observations. In mod-
els with dependent observations quite different rates may pertain.
14.35 Example (Bra n eh ing). Consider the Galton-Watson branching process, discussed
in Example 9.10. If the offspring distribution has mean IL (e) larger than 1, then the parameter
is estimable at the exponential rate IL (e)n. This is also the right rescaling rate for defining
asymptotic power functions. D
Notes
Apparently, E.lG. Pitman introduced the efficiencies that are named for him in an unpub-
lished set of lecture note s in 1949. A published proof of a slightly more general result can
be found in [109].
Cramer [26] was interested in preciser approximations to probabilities of large deviations
than are presented in this chapter and obtained the theorem under the condition that the
moment-generating function is finite on JR. Chemoff [20] proved the theorem as presented
here, by a different argument. Chernoff used it to study the minimum weighted sums of
error probabilities of tests that reject for large values of a mean and showed that, for any
O < Jr < 1,
log inf(nPo(Y > t) + (1 - n)P 1 (Y < t))
n t
--* inf inf ( Ko (u) - u t) v inf ( K I (u) - u t ) .
EaYl <t<Elf l u u
Furthermore, for Y the likelihood ratio statistic for testing Po versus Pl, the right side of
this display can be expressed in the Hellinger integral of the experiment (Po, Pl) as
inf l Og f dP dPll-u.
O<u<l
Thus, this expression is a lower bound for the lim inf n --+ oo n-Ilog(a n + f3n) for an and f3n the
error probabilities of any test of Po versus Pl. That the Bahadur slope of Neyman-Pearson
tests is twice the Kullback - Leibler divergence (Example 14.26) is essentially known as
Stein' s lemma and is apparently among those results by Stein that he never cared to publish.
A first version of Sanov's theorem was proved by Sanov in 1957. Subsequently, many
authors contributed to strengthening the result, the version presented here being given in
[65]. Large-deviation theorems are subject of current research by probabilists, particularly
with extensions to more complicated objects than sums of independent variables. See [31]
and [32]. For further information and reference s concerning applications in statistics, we
refer to [4] and [61], as well as to Chapters 8,16, and 17.
For applications and extensions of the results on rescaling rates, see [37].
PROBLEMS
1. Show that the power function of the Wilcoxon two sample test is monotone under shift oflocation.
2. Let XI, . . . , X n be arandom sample from the N (fl, () 2 ) -distribution, where () 2 is known. A test
- -
for Ho : fl == O against HI : fl > O can be based on either X / () Of X / S. Show that the asymptotic
214
Relative Efficiency of Tests
relative efficiency of the two sequences of tests is 1. Does it make a difference whether normal
or t-critical values are used?
3. Let XI,..., X n be arandom sample from a density f(x - e) where f is symmetric about
zero. Calculate the relative efficiency of the t-test and the test that rejects for large values of
LLi <j 1 {Xi + X j > O} for f equal to the logistic, normal, Laplace, and uniform shapes.
4. Calculate the relative efficiency of the van der Waerden test with respect to the t -test in the
two-sample problem.
5. Calculate the relative efficiency of the tests based on Kendall's i and the sample correlation
coefficient to test independence for bivariate normal pairs of observations.
6. Suppose ep : :F 1-+ IR and 1/1 : :F 1-+ IR k are arbitrary maps on an arbitrary set :F and we wish to
find the minimum value of ep over the set {f E :F: 1/1(f) == O}. Ifthe map f 1-+ ep(f) +a T 1/1(f)
attains its minimum over :F at fa, for each fixed a in an arbitrary set A, and there exists ao E A such
that 1/1 (fao) == O, then the desired minimum value is ep (fao)' This is a rather trivial use of Lagrange
multipliers, but it is helpful to solve the next problems. (ep (fao) == ep (fao) + a6 1/1 (fao) is the
minimum of ep (f) + a6 1/1 (f) over :F and hence smaller than the minimum of ep (f) + a6 1/1 (f)
over {f E :F: 1/1(f) == O}.)
7. Show that 4f(0)2 1 y2 f(y) dy > 1/3 for every probability density f that has its mode at O.
(The minimum is equal to the minimum of 4 1 y2 f(y) dy over all probability densities f that
are bounded by 1.)
8. Show that 12(1 f2(y) dy)2 1 y2 f(y) dy > 108/125 for every probability density f with mean
zero. (The minimum is equal to 12 times the minimum of the square of ep(f) == 1 f2(y) dy over
all probability densities with mean O and variance 1.)
9. Study the asymptotic power function of the sign test if the observations are a sample from
a distribution that has apositive mass at its median. Is it good or bad to have a nonsmooth
distribution?
10. Calculate the Hellinger and total variation distance between two uniform U[O, e] measures.
11. Calculate the Hellinger and total variation distance between two normal N(JL, (J2) measures.
12. Let XI, . . . , X n be a sample from the uniform distribution on [-e, e].
(i) Calculate the asymptotic power functions of the tests that reject Ha : e == eo for large values
of X(n), X(n) V (-X(l)) and X(n) - X(l).
(ii) Calculate the asymptotic relative efficiencies of these tests.
13. Iftwo sequences of test statistics satisfied (14.4) forevery en + O, but with normingrate na instead
of ,.jn, how would Theorem 14.19 have to be modified to find the Pitman relative efficiency?
15
Efficiency of Tests
It is shown that, given converging experiments, every limiting power
function is the power function of a test in the limit experiment. Thus,
uniformly most powerful tests in the limit experiment give absolute upper
bounds for the power of a sequence of tests. In normal experiments such
uniformly most powerful tests exist for linear hypotheses of codimension
one. The one-sample location problem and the two-sample problem are
discussed in de ta i!, and appropriately designed (signed) rank tests are
shown to be asymptotically optimal.
15.1 Asymptotic Representation Theorem
A randomized test (or test function) C/J in an experiment (X, A, P h : h E H) is a measurable
map C/J : X r--+ [O, 1] on the sample space. The interpretation is that if x is observed, then
a nulI hypothesis is rejected with probability ep (x). The power function of a test ep is the
function
h r--+ 7f (h) == Ehc/J (X).
This gives the probabilities that the nun hypothesis is rejected. A test is of level ex for testing
a nulI hypothesis Ha if its size sup { 7f (h) : h E Ha} does not exceed ex. The quality of a
test can be judged from its power function, and classical testing theory is aimed at finding,
among the tests of level ex, a test with high power at every alternative.
The asymptotic quality of a sequence of tests may be judged from the limit of the
sequence of local power functions. If the tests are defined in experiments that converge
to alimit experiment, then a pointwise limit of power functions is necessarily a power
function in the limit experiment. This follows from the following theorem, which specializes
the asymptotic representation theorem, Theorem 9.3, to the testing problem. Applied to
the special case of the local experiments En == (Pen+h/.Jii: h E JRk) of a differentiable
parametric model as considered in Chapter 7, which converge to the Gaussian experiment
(N (h, le-I), h E k), the theorem is the paralleI for testing of Theorem 7.10.
15.1 Theorem. Let the sequence of experiments En == (Pn,h : h E H) converge to a domi-
nated experiment E == (P h : h EH). Suppose that a sequence of power functions 7f n of
tests in En converges pointwise: 7f n (h) -+ 7f(h), for every h and some arbitrary function
7f. Then 7f is a power function in the limit experiment: There exists a test C/J in E with
7f(h) == Ehc/J (X) for every h.
215
216
Efficiency of Tests
Proof. We give the proof for the special case of experiments that satisfy the following
assumption: Every sequence of statistics Tn that is tight under every given parameter h
possesses a subsequence (not depending on h) that converges in distribution to alimit under
every h. See problem 15.2 for a method to extend the proof to the general situation.
The additional condition is valid in the case of local asymptotic normality. With the
notation of the proof of Theorem 7.10, we argue first that the sequence (Tn, n) is uni-
formly tight under h == O and hence possesses a weakly convergent subsequence by
Prohorov's theorem. Next, by the expansion of the likelihood and Slutsky's lemma, the se-
quence (Tn, log dPn,h/dPn,a) converges under h == O along the same sequence, for every
h. Finally, we conclude by Le Cam' s third lemma that the sequence Tn converges under h,
along the subsequence.
Let epn be tests with power functions 1Tn. Because each epn takes its values in the com-
pact interval [O, 1], the sequence of random variables epn is certainly uniformly tight. By
assumption, there exists a subsequence of {n} along which epn converges in distribution un-
der every h. Thus, the assumption of the asymptotic representation theorem, Theorem 9.3
or Theorem 7.10, is satisfied along some subsequence of the statistics epn. By this theorem,
there exists a randomized statistic T == T (X, U) in the limit experiment such that epn T
along the subsequence, for every h. The randomized statistic may be assumed to take its
values in [O, 1]. Because the epn are uniformly bounded, Ehepn ---+ Eh T. Combination with
the assumption yields 1T(h) == EhT for every h. The randomized statistic T is not a test
function (it is a "doubly randomized" test). However, the test ep (x) == E(T (X, U) IX == x)
satisfies the requirements. .
The theorem suggests that the best possible limiting power function is the power function
of the best test in the limit experiment. In classical testing theory an "absolutely best" test
is defined as a uniformly most powerful test of the required level. Depending on the
experiment, such a test may or may not exist. If it does not exist, then the classical solution
is to find a uniformly most powerful test in arestricted class, such as the class of all
unbiased or invariant tests; to use the maximin criterion; or to use a conditional test. In
combination with the preceding theorem, each of these approaches leads to a criterion for
asymptotic quality. We do not pursue this in detail but note that, in general, we would avoid
any sequence of tests that is matched in the limit experiment by a test that is considered
suboptimal.
In the remainder of this chapter we consider the implications for locally asymptotically
normal models in more detai1. We start with reviewing testing in normallocation models.
15.2 Testing Normal Means
Suppose that the observation X is Nk(h, )-distributed, for a known covariance matrix
and unknown mean vector h. First consider testing the nulI hypothesis Ha : eT h == O versus
the alternative HI : eT h > O, for a known vector e. The "naturaI" test, which rejects Ha for
large values of eT X, is uniformly most powerfu1. In other words, if 1T is a power function
such that 1T(h) < a for every h with eT h == O, then for every h with eT h > O,
( eT h )
1T(h) < Ph(eTX > Za -J eT e) == 1 - <I> Za - .
-J eT e
15.2 Testing Normal Means
217
15.2 Proposition. Suppose that X be Nk(h, )-distributed for a known nonnegative-
definite matrix , an d let e be afixed veetor with eT e > O. Then the test that rejeets Ha
if eT X > Za v' eT e is uniformly most powerful at level ex for testing Ha : eT h == O versus
HI : eT h > O, based on X.
Proof. FixhI with eT hI >0. Defineh a ==h 1 - (eThl/eT e) e. TheneTha==O. By the
Neyman-Pearson lemma, the most powerful test for testing the simple hypotheses Ha : h ==
ha and HI : h == hI rejects Ha for large values of
dN(hI, ) eThI T 1 (e T h 1 )2
log (X) == e X - - .
dN(ho, ) eT e 2 eT e
This is equivalent to the test that rejects for large values of eT X. More precisely, the most
powerfullevel ex test for Ha : h == ha versus HI : h == hI is the test given by the proposition.
Because this test does not depend on ha or hI, it is uniformly most powerful for testing
Ha: eTh == O versus HI : eTh > O. .
The natural test for the two-sided problem Ha : eT h == O versus HI : eT h i=- O rejects the
null hypothesis for large values of I eT X I. This test is not uniformly most powerful, because
its power is dominated by the uniformly most powerful tests for the two one-sided alterna -
tives whose union is HI. However, the test with critical regi on {x : leT x I > Za/2 v' eT e}
is uniformly most powerful among the unbiased level ex tests (see problem 15.1).
A second problem of interest is to test a simple nulI hypothesis Ha : h == O versus the
alternative HI : h :j=. O. If the parameter set is one-dimensional, then this reduces to the
problem in the preceding paragraph. However, if e is of dimension k > 1, then there exists
no uniformly most powerful test, not even among the unbiased tests. A variety of tests are
reasonable, and whether a test is "good" depends on the alternative s at which we desire high
power. For instance, the test that is most sensitive to detect the alternative s such that eT h > O
(for a given e) is the test given in the preceding theorem. Probably in most situations no
particular "direction" is of special importance, and we would use a test that distributes the
power over all directions. It is known that any test with as critical region the complement
of a closed, convex set C is admissible (see, e.g., [138, p. 137]). In particular, complements
of closed, convex, and symmetric sets are admissible critical regions and cannot easily be
ruled out a priori. The shape of edetermines the power function, the directions in which
e extends little receiving large power (although the power also depends on ).
The most popular test rejects the null hypothesis for large values of X T -1 X. This test
arises as the limit version of the Wald test, the score test, and the likelihood ratio test. One ad-
vantage is a simple choice of critical values, because X T -1 X is chi square-distributed with
k degrees of freedom. The power function of this test is, with Z a standard normal vector,
n(h) == Ph(XT -1 X > Xl,a) == p(IIZ + -I/2hI12 > Xl,a).
By the rotational symmetry of the standard normal distribution, this depends only on the non-
eentrality parameter II -1/2 h II. The power is relatively large in the directions h for which
II -1/2h II is large. In particular, it increases most steeply in the direction of the eigenvector
corresponding to the smallest eigenvalue of . N ote that the test does not distribute the
power evenly, but dependent on . Two optimality properties of this test are given in
problems 15.3 and 15.4, but these do not really seem convincing.
218
Efficiency of Tests
Due to the lack of an acceptable optimal test in the limit problem, a satisfactory asymp-
totic optimality theory of testing simple hypotheses on multidimensional parameters is
impossible.
15.3 Local Asymptotic Normality
A normallimit experiment arises, among others, in the situation of repeated sampling from
a differentiable parametric model. If the model (Pe : 8 E 8) is differentiable in quadratic
mean, then the local experiments converge to a Gaussian limit:
(P:O+h/-fii: h E Jl n -7 (N(h, 10;,1): h E JRk).
A sequence of power functions 8 1---+ n n (8) in the original experiments induces the sequence
of power functions h 1---+ nn (8 0 + h/ ) in the local experiments. Suppose that nn (8 0 +
h I ) n (h) for every h and some function n. Then, by the asymptotic representation
theorem, the limit n is a power function in the Gaussian limit experiment.
Suppose for the moment that 8 is real and that the sequence nn is of asymptotic level ex
for testing Ho : 8 < 8 0 versus HI : 8 > 8 0 . Then n(O) == limn n (80) < ex and hence n
corresponds to a level ex test for Ho : h == O versus HI : h > O in the limit experiment. It must
be bounded above by the power function of the uniformly most powerfullevel ex test in the
limit experiment, which is given by Proposition 15.2. Conclude that
lim nn ( e o + ) < 1 - <1> (Za - h-lf;;), every h > O.
n oo '\! n
(Apply the proposition with c == 1 and:E == le I.) We have derived an absolute upper bound
on the local asymptotic power of level ex tests.
In Chapter 14 a sequence ofpower functions such that nn (8 0 + h/ ) 1- <1> (Za -hs)
for every h is said to have slope s. It follows from the present upper bound that the square
root A of the Fisher information is the largest possible slope. The quantity
leo
s2
is the relative efficiency of the best test and the test with slope s. It can be interpreted as
the number of observations needed with the given sequence of tests with slope s divided by
the number of observations needed with the best test to obtain the same power.
With a bit of work, the assumption that n n (8 0 + h I ) converges to a limit for every h
can be removed. Also, the preceding derivati on does not use the special structure of i.i.d.
observations but only uses the convergence to a Gaussian experiment. We shall rederive the
result within the context of local asymptotic norrnality and also indie ate how to eonstruet
optimal tests.
Suppose that at "time" n the observation is distributed according to a distribution Pn,e with
parameter ranging over an open subset 8 ofIR k . The sequenee of experiments (Pn,e : 8 E 8)
is loeally asymptotically normal at 8 0 if
d P e - I h 1
I n, o+rn T T
O g == h n e o - - h le o h + O p e (1),
d P n, e o ' 2 n, O
(15.3)
15.3 Local Asymptotic Normality
219
for a sequence of statistics n,rJo that converges in distribution under ()o to a normal
Nk(O,leo)-distribution.
15.4 Theorem. Let 8 C}Rk be open and let 1/1' : 8 r--+ JR be differentiable at ()o, with
nonzero gradient 1/1' e o and such that 1/1' (()o) == O. Let the sequence of experiments
(Pn,e: () E 8) be locally asymptotically normal at (Jo with nonsingular Fisher informa-
tion, for constants r n -* 00. Then the power functions () r--+ Jr n (()) of any sequence of
level a tests for testing Ha : 1/f(e) < O versus HI : 1/1'(()) > O satisfy, for every h such that
1/1'e o h > O,
lim sup Jr n ( ()O + !!.- ) < 1 - <p ( Za -
n ---H)O r n
1/1' e o h )
. 1 . T
1/f eo I e 1/1' eo
15.5 Addendum. Let Tn be statistics such that
"il' I-I/),.
7' 'Peo e o n,eo (1)
1n== +oPe'
. l' T n, O
1/1' eo I e 1/1' eo
Then the sequence oftests that rejectforvalues ofTn exceeding Za is asymptotically optimal
in the sense that the sequence P eo+r;l h (Tn > Za) converges to the right side of the preceding
display, for every h.
Proofs. The sequence of localized experiments (Pn,eo+r;lh : h E JRk) converges by Theo-
rem 7.10, or Theorem 9.4, to the Gaussian location experiment (Nk(h, le-I) : h E JRk).
. o
Fix some hI such that 1/1'e o hI> O, and a subsequence of {n} along which the lim sup
Jr ((Ja + hI / r n) is taken. There exists a further subsequence along which Jr n (()o + r;; 1 h)
converges to alimit Jr(h) for every h E JRk (see the proof of Theorem 15.1). The function
h r--+ Jr (h) is a power function in the Gaussian limit experiment. For 1/1' e o h < O, we have
1/1' (()o + r;;1 h) == r;;l (feoh+o(l)) < O eventually, whence Jr(h) < lin: sup Jr n (()o + r;;1 h) <
a. By continuity, the i equality Jr(h) < a e tends to all h such that 1/1'eoh < O. Thus, Jr is of
level a for testing Ho : 1/1'e o h < O versus HI : 1/f e o h > O. Its power function is bounded above by
the power function of the uniformly most powerful test, which is given by Proposition 15.2.
This concludes the proof of the theorem.
The asymptotic optimality of the sequence Tn follows by contiguity arguments. We start
by noting that the sequence ( n,eo' n,eo) converges under ()o in distribution to a (degenerate)
normal vector ( , ). By Slutsky's lemma and local asymptotic normality,
( /),. lo d Pn, eo+r; 1 h ) ( h T _ 1 h TIh )
n, eo ' g d P , 2 e o
n,eo
N ( ( -4h Ieah ) , (h e;ea
leo h ) )
h T Ieo h .
By Le Cam's third lemma, the sequence n,eo converges in distribution under ()o + r;;1 h to a
N (Ieah, leo)-distribution. Thus, the sequence Tn converges under ()o + r;;1 h in distribution
to a normal distribution with mean feoh/(feole lf )1/2 and variance 1. .
220
Efficiency of Tests
The point e a in the preceding theorem is on the boundary of the nulI and the alter-
native hypotheses. If the dimension k is larger than 1, then this boundary is typicalIy
(k - 1 )-dimensional, and there are many possible values for e a . The upper bound is valid
at every possible choice.
If k == 1, the boundary point e a is typically unique and hence known, and we could use
-1/2 . 1 f
Tn == leo n,eo to construct an ophma sequence o tests for the problem Ha : e == e a .
These are known as score tests.
Another possibility is to base a test on an estimator sequence. Not surprisingly, efficient
estimators yield efficient tests.
15.6 Example (Wald tests). Let XI,..., X n be arandom sample in an experiment
(Pe : e E 8) that is differentiable in quadratic mean with nonsingular Fisher informa-
tion. Then the sequence of local experiments (P e n + h /-J1i : h E JRk) is locally asymptotically
normal with r n == -Jli, le the Fisher information matrix, and
1 n .
n,e == -Jli L (Xi).
n. 1
l=
A sequence of estimators en is asymptotically efficient for estimating e if (see Chapter 8)
,jfi(e n - e) == le- 1 n,e +oPe(l).
Under regularity conditions, the maximum likelihood estimator qualifies. Suppose that
e 1---+ le is continuous, and that 1/1 is continuously differentiable with nonzero gradient.
Then the sequence of tests that reject Ha : 1/1 (e) < O if
,jfi 1fr (en) > Za ..fr eJt ..fr l
is asymptoticalIy optimal at every point e a on the boundary of Ha. Furthermore, this seq-
uence of tests is consistent at every e with 1/1(e) > O.
These assertions follow from the preceding theorem, upon using the delta method and
Slutsk:y' s lemma. The resulting tests are called Wald tests if en is the maximum likelihood
estimator. D
15.4 One-Sample Location
Let XI, . . . , X n be a sample from a density f (x - e), where f is symmetric about zero
and has finite Fisher information for location I f' It is required to test Ha : e == O versus
HI : e > O. The density f may be known or (partially) unknown. For instance, it may be
known to belong to the normal scale family.
For fixed .f, the sequence of experiments (Il7=1 f(Xi - e) : e E IR) is locally asymptot-
ically normal at e == O with n,a == -n- 1 / 2 :L7=1 (f' / f)(X i ), norming rate -Jli, and Fisher
information I f. By the results of the preceding section, the best asymptotic level ex power
function (for known f) is
1 - cp(Za - hjff).
15.4 One-Sample Location
221
This function is an upper bound for lim sup Jr n (h / -Jfi), for every h > O, for every sequence
of level ex power functions. Suppose that Tn are statistics with
lIn J'
Tn == - r:; rT::" L - (X i) + O Po (1) .
v n y'Ili=l J
(15.7)
Then, according to the second assertion of Theorem 15.4, the sequence of tests that reject
the nulI hypothesis if Tn > Za attains the bound and hence is asymptoticalIy optimal. We
shalI discuss several ways of constructing test statistics with this property.
If the shape of the distribution is completely known, then the test statistics Tn can simply
be taken equal to the right side of (15.7), without the remainder term, and we obtain the
score test. It is more realistic to assume that the underlying distribution is only known up
to scale. If the underlying density takes the form J (x) == Ja (x / a) / a for a known density
Ja that is syrnmetric about zero, but for an unknown scale parameter a, then
J' 1 J x
y(x) = a lo ( a )'
1
II == 2 110 '
a
1 J' 1 J x
jlf Y(X) = /l!o lo ( a ).
15.8 Example (t-test). The standard normal density Ja possesses score function J / Ja
(x) == - x and Fisher information I lo == 1. Consequently, if the underlying distribution is
normal, then the optimal test statistics should satisfy Tn == -JfiX n / a + o Po (n -1/2). The
t-statistics Xn/ Sn fulfill this requirement. This is not surprising, because in the case of nor-
malIy distributed observations the t -test is uniformly most powerful for every finite n and
hence is certainly asymptotically optimal. D
The t -statistic in the preceding example simply replaces the unknown standard deviation
a by an estimate. This approach can be followed for most scale families. Under some regu-
larity conditions, the statistics
Tn = - t / ( Xi )
-Jfi /l!o i=l Ja an
should yield asymptotically optimal tests, given a consistent sequence of scale esti-
mators an.
Rather than using score-type tests, we could use a test based on an efficient estimator for
the unknown symmetry point and efficient estimators for possible nuisance parameters, such
as the scale - for instance, the maximum likelihood estimators. This method is indicated in
general in Example 15.8 and leads to the Wald test.
Perhaps the most attractive approach is to use signed rank statistics. We summarize some
definitions and conclusions from Chapter 13. Let R: 1 , . . . , R: n be the ranks of the absolute
values IX 11, . . . , I X n I in the ordered sample of absolute values. A linear signed rank statistic
takes the form
1 n
Tn == -Jfi an R sign(X i ),
n 'ni
i=l
for given numbers anI, . . . , ann, which are called the scores of the statistic. Particular
examples are the Wilcoxon signed rank statistic, which has scores ani == i, and the sign
statistic, which corresponds to scores ani == 1. In general, the scores can be chosen to weigh
222
Efficiency of Tests
the influence of the different observations. A convenient method of generating scores is
through a fixed function ep : [O, 1] r-+ IR, by
ani == Eep (Un(i))'
(Here U n (1), . . . , Un(n) are the order statistics of arandom sample of size n from the uniform
distribution on [0,1].) Under the condition that J cjJ2(u) du < 00, Theorem 13.18 shows
that, under the nulI hypothesis, and with F+ (x) == 2F (x) - 1 denoting the distribution
function of IX 11,
1 n
Tn = .J11 8<P( p+ (IXiI)) sign(X i ) + OPO (1).
Because the score-generating function ep can be chosen freely, this alIows the construction
of an asymptoticalIy optimal rank statistic for any given shape f. The choice
1 fl
ep(u) == - rr=- -((F+)-l(u)).
yIJ f
(15.9)
yields the locally most powerful scores, as discussed in Chapter 13. Because fl j f (Ix I) sign
(x) == fl j f (x) by the symmetry of f, it follows that the signed rank statistics Tn satisfy
(15.7). Thus, the 10calIy most powerful scores yield asymptoticalIy optimal signed rank
tests. This surprising result, that the class of signed rank statistics contains asymptoticalIy
efficient tests for every given (symmetric) shape of the underlying distribution, is sometimes
expressed by saying that the signs and absolute ranks are "asymptotically sufficient" for
testing the location of a symmetry point.
15.10 Corollary. Let Tn be the simple linear signed rank statistic with scores ani ==
Eep (Un(i)) generated by the function ep defined in (15.9). Then Tn satisfies (15.7) and hence
the sequence of tests that reject Ho : e == O if Tn > Za is asymptotically optimal at e == o.
Signed rank statistics were originalIy constructed because of their attractive property of
being distribution free under the nulI hypothesis. Apparently, this can be achieved without
losing (asymptotic) power. Thus, rank tests are strong competitors of classical parametric
tests. Note also that signed rank statistics automaticalIy adapt to the unknown scale: Even
though the definition of the optimal scores appears to depend on f, they are actually identical
for every member of a scale family f(x) == fo(xja)ja (since (F+)-l (u) == a(Fo+)-l (u)).
Thus, no auxiliary estimate for a is necessary for their definition.
15.11 Example (Laplace). The sign statistic Tn == n- 1 / 2 L7=1 sign(X i ) satisfies (15.7)
for f equal to the Laplace density. Thus the sign test is asymptotically optimal for testing
location in the Laplace scale family. O
15.12 Example (Normal). The standard normal density has score function for location
f j fo(x) == -x and Fisher information I Ja == 1. The optimal signed rank statistic for the
normal scale family has score-generating function
<p (u) = E(<I>+) -I (Un(i») = E<I>-I ( Un(i + 1 ) <1>-1 ( 2n 2 + ).
15.5 Two-Sample Problems
223
We eonclude that the corresponding sequence of rank tests has the same asymptotic slope as
the t-test ifthe underlying distribution is normal. (For other distributions the two sequences
of tests have different asymptotic behavior.) D
Even the assumption that the underlying distribution of the observations is known up to
scale is often unrealistic. Because rank statistics are distribution- free under the null hypo-
thesis, the level of a rank test is independent of the underlying distribution, which is the best
possible protection of the level against misspecification of the model. On the other hand, the
power of a rank test is not necessarily robu st against deviations from the postulated model.
This might lead to the use of the best test for the wrong model. The dependence of the
power on the underlying distribution may be relaxed as well, by a procedure known as
adaptation. This entails estimating the underlying density from the data and next using
an optimal test for the estimated density. A remarkable fact is that this approach can be
completely successful: There exist test statistics that are asymptotically optimal for any
shape f. In fact, without prior knowledge of f (other than that it is symmetric with finite
and positive Fisher information for location), estimators en and In can be constructed such
that, for every e and f,
" lIn fl
vYt(e n - e) == - /;; - 1 L - f (Xi - e) + oP e (l);
V n f i=I
Pe
In 'v'-t IJ.
We give such a construction in section 25.8.1. Then the test statistics Tn == VYt en I;; /2 satisfy
(15.7) and hence are asymptotically (1ocally) optimal at e == O for every given shape f.
Moreover, for every e > O, and every f,
Pe(Tn > zQJ == Pe(vYt(e n - e) > Za I n-I/2 - vYte) -+ 1.
Renee, the sequence of tests based on Tn is also consistent at every (e, f) in the alternative
hypothesis HI : e > o.
15.5 Two-Sample Problems
Suppose we observe two independent random samples XI, . . . , X m and YI, . . . , Y n from
densities p J.L and q v, respectively. The problem is to test the null hypothesis Ha : 1) < f.-L versus
the alternative HI : V > f.-L. There may be other unknown parameters in the model besides f.-L
and 1), but we shall initially ignore such "nuisanee parameters" and parametrize the model
by (IL, v) E IR? Null and alternative hypotheses are shown graphically in Figure 15.1. We
let N == m + n be the total number of observations and assume that m / N -+ A as m, n -+ 00.
15.13 Example (Testing shift). If PJ.L (x) == f (x - f.-L) and qv (y) == g(y - 1)) for two densi-
ties f and g that have the same "location," then we obtain the two-sample location problem.
The alternative hypothesis asserts that the second sample is "stochastically larger." D
The alternative s of greatest interest for the study of the asymptotic performance of
tests are sequences (f.-L N, VN) that converge to the boundary between null and alternative
hypotheses. In the study of relative efficiency, in Chapter 14, we restricted ourselves to
224
Efficžency of Tests
v
HI
\
Ha
Figure 15.1. Null and alternative hypothesis.
vertical perturbations (e, e + hl,JR). Here we shall use the sequences (e + gl,JR, e +
h I ,JR), which approach the boundary in the direction of a general vec tor (g, h).
If both p J-L and qv define differentiable models, then the sequence of experiments (P;: Q9
P:: (f.-t, v) E IR?) is locally asy ptotically normal with norming rate,JR. If the score
functions are denoted by KJ-L and fv, and the Fisher informations by IJ-L and iv, respectively,
then the parameters of local asymptotic normality are
n, (J-L, v) ==
v0: m .
vm LKJ-L(X ž )
m. 1
l=
,J I - An.
lv(Yj)
( AI J-L
I (J-L, v) == O
(1 _D)")1v ) .
The corresponding limit experiment consists of observing two independent normally dis-
tributed variables with means g and h and variances A -11;:1 and (1- A)-1 lv- 1 , respectively.
15.14 Corollary. Suppase that the lnodels (PJ-L : f.-t E JR) and (Qv : V E JR) are differen-
tiable in quadratic mean, and let m, n ---+ 00 such that ml N ---+ A E (O, 1). Then the
power.functions of any sequence of level ex testsfor Ha : V == f.-t satisfies, for every f.-t and for
every h > g,
( g h ) ( A(1 - A)IJ-LIJ-L )
lim sup 7Tm,n f.-t + ri:T ' f.-t + ri:T < 1 - <I:> Za - (h - g)
n,m-+oo y N y N AIJ-L + (1 - A)lJ-L .
Proof. This is a special case of Theorem 15.4, with 1/1' (f.-t, v) == v - f.-t and Fisher in-
formation matrix diag (AI 11-' (1 - A) I J-L)' It is slightly different in that the null hypothesis
Ha : 1/1' (e) == O takes the form of an equality, which gives a weaker requirement on the
sequence Tn. The proof goe s through because of the linearity of 1/1' . .
15.5 Two-Sample Problems
225
It follows that the optimal slope of a sequence of tests is equal to
Sopt (J-L) ==
A(l - A)IfLJJ1-
AIJ-l + (1 - A)JfL
The square of the quotient of the actual slope of a sequence of tests and this number is a
good absolute measure of the asymptotic quality of the sequence of tests.
According to the second assertion of Theorem 15.4, an optimal sequence of tests can be
based on any sequence of statistics such that
( lIn. lIm )
TN = Sopt(/L) {.; L.€JL(Yj) - -JI r;;;;; LKJL(X i ) + opel).
,J I - AJfL V n j=1 AlfL V m i=1
(The multiplicative factor Sopt(J-L) ensures that the sequence TN is asymptotically normally
distributed with variance 1.) Test statistics with this property can be constructed using a
variety of methods. For instance, in rnany cases we can use asymptotically efficient esti-
mators for the parameters J-L and 1), combined with estimators for possible nuisance param-
eters, along the lines of Exarnple 15.6.
If p J-l == q fL == i fL are equal and are densities on the real line, then rank statistics are
attractive. Let R N1 , . . . , RNN be the ranks of the pooled sample XI, . . . , X m , YI, . . . , Y n .
Consider the two-sample rank statistics
1 N
TN == ri:i L aN,RNi'
V N i=m+l
a N i == EcjJ ( U N (i) ) ,
for the score generating function
1 . ( -1 )
<fJ(u) = ..j)'(l _ ).) jT/JL FJL (u) .
Up to a constant these are the locally most powerful scores introduced in Chapter 13. By
Theorem 13.5, because a N == 101 cjJ(u) du == O,
1 ( Im 1 n )
TN == -- ,J I - A- LffL(X i ) - Vi- LffL(Yj) + OPjL (1).
-Ji fL vin i=1 j=1
Thus, the locally most powerful rank statistics yield asymptotically optimal tests. In general,
the optimal rank test depends on J-L, and other parameters in the model, which rnust be
estimated from the data, but in the most interesting cases this is not necessary.
15.15 Example (Wilcoxon s tatistic ). For i fL equal to the logistic density with mean J-L,
the scores aN,i are proportional to i. Thus, the Wilcoxon (or Mann-Whitney) two-sample
statistic is asymptotically uniformly most powerful for testing a difference in location
between two samples from logistic densities with different means. O
15.16 Example (Log rank test). The log rank test is asymptotically optimal for testing
proportional hazard altematives, given any baseline distribution. O
226
Efficiency of Tests
Notes
Absolute bounds on asymptotic power functions as developed in this chapter are less known
than the absolute bounds on estimator sequences given in Chapter 8. Testing problems were
nevertheless an important subject in Wald [149], who is credited by Le Cam for having first
conceived of the method of approximating experiments by Gaussian experiments, albeit
in a somewhat different way than later developed by Le Cam. From the point of view
of statistical decision theory, there is no difference between testing and estimating, and
hence the asymptotic bounds for tests in this chapter fit in the general theory developed
in [99]. Wald appears to use the Gaussian approximation to transfer the optimality of the
likelihood ratio and the Wald test (that is now named for him) in the Gaussian experiment
to the sequence of experiments. In our discussion we use the Gaussian approximation to
show that, in the multidimensional case, "asymptotic optimality" can only be defined in a
somewhat arbitrary manner, because optimality in the Gaussian experiment is not easy to
define. That is a difference of taste.
PROBLEMS
1. Consider the two-sided testing problem Ha : c T h == O versus HI : c T h =I O based on an Nk(h, 2:)-
distributed observation X. A test for testing Ha versus HI is called unbiased if SUPhEHo Te (h) <
infhEH1 Te(h). The test that rejects Ha for large values of IcT XI is uniformly most powerful among
the unbiased tests. More precisely, for every power function Te of a test based on X the conditions
Te (h) < ex if h T c == O
and
Te (h) > ex if h T c =I O,
imply that, for every c T h =I O,
( cT h ) ( cT h )
Te(h) < p(lc T XI > Za/2 J CTbC) == 1 - ep Za/2 - + 1 - ep Za/2 + .
cTbC cTbC
Formulate an asymptotic upper bound theorem for two-sided testing problems in the spirit of
Theorem 15.4.
2. (i) Show that the set of power functions h r-+ Te n (h) in a dominated experiment (Ph : h E H) is
compact for the topology of pointwise convergence (on H).
(ii) Give a full proof of Theorem 15.1 along the following lines. First apply the proof as given
for every finite subset I eH. This yields power functions Te I in the limit experiment that
coincide with Te on I.
3. Consider testing Ha: h == O versus HI: h =I O based on an observation X with an
N(h, 2:)-distribution. Show that the testing problem is invariant under the transformations x r-+
b 1/2 O b -1/2 x for O ranging over the orthonormal group. Find the best invariant test.
4. Consider testing Ha : h == O versus HI : h =I O based on an observation X with an N(h, 2:)-
distribution. Find the test that maximizes the minimum power over {h : II b -1/2 hil == c}. (By the
Hunt-Stein theorem the best invariant test is maximin, so one can apply the preceding problem.
Alternatively, one can give a direct derivati on along the following lines. Let Te be the distribution
of b 1 / 2 U if U is uniformly distributed on the set {h: Ilhll == cl. Derive the Neyman-Pearson
test for testing Ha : N(O, b) versus HI : f N(h, b) dTe(h). Show that its power is constant on
{h : 112: -1/2 hil == c}. The minimum power of any test on this set is always smaller than the average
power over this set, which is the power at f N(h, 2:) dTe(h).)
16
Likelihood Ratio Tests
The critical values of the likelihood ratio test are usually based on an
asymptotic approximation. We derive the asymptotic distribution of the
likelihood ratio statistic and investigate its asymptotic quality through its
asymptotic power function and its Bahadur efficiency.
16.1 Introduction
Suppose that we observe a sample XI, . . . , X n from a density pe, and wish to test the
nulI hypothesis Ho : e E 8 0 versus the alternative HI : e E 8 1 . If both the nulI and the
alternative hypotheses consist of single points, then a most powerful test can be based on
the log likelihood ratio, by the Neyman-Pearson theory. If the two points are e o and el,
respectively, then the optimal test statistic is given by
1 IT7=lPel (Xi)
og IT n .
i=l pe o (Xi)
For certain special models and hypotheses, the most powerful test turns out not to depend
on el, and the test is uniformly most powerful for a composite hypothesis 8 1 . Sometimes
the nulI hypothesis can be extended as well, and the testing problem has a fully satisfac-
tory solution. Unfortunately, in many situations there is no single best test, not even in an
asymptotic sense (see Chapter 15). A variety of ideas lead to reasonable tests. A sensible
extension of the idea behind the N eyman- Pearson theory is to base a test on the log likelihood
ratio
A - _ 1 sUPeE8\ IT7=lPe(X i )
n - og n .
sUPeE8 0 ITi=l pe (Xi)
The single points are replaced by maxima over the hypotheses. As before, the nulI hypoth-
esis is rejected for large values of the statistic.
Because the distributional properties of An can be somewhat complicated, one usually
replaces the supremum in the numerator by a supremum over the whole parameter set
8 == 8 0 U 8 1 . This changes the test statistic only if An < O, which is inessential, because
in most cases the critical value will be positive. We study the asymptotic properties of the
227
228
Likelihood Ratio Tests
(log) likelihood ratio statistic
A - 21 sUPeE8 fl7=IPe (Xi) 2(A O)
n - og n == - n V .
SUPeE8 0 fli=I Pe (Xi)
The most important conclusion of this chapter is that, under the nulI hypothesis, the sequence
An is asymptoticalIy ehi squared-distributed. The main conditions are that the model is
differentiable in () and that the nulI hypothesis 8 0 and the full parameter set 8 are (locally)
equal to linear spaces. The number of degrees of freedom is equal to the difference of the
(local) dimensions of 8 and 8 0 . Then the test that rejects the null hypothesis if An exceeds
the upper a-quantile of the ehi-square distribution is asymptoticalIy of level a. Throughout
the chapter we as sume that 8 c }Rk.
The "local linearity" of the hypotheses is essential for the ehi -square approximation,
which fails already in a number of simple examples. An open set is certainly locally linear
at every of its points, and so is a relatively open subset of an affine subspace. On the
other hand, a half line or space, which arises, for instance, if testing a one-sided hypothesis
Ho : /-Le < O, or a ball Ho : II () II < 1, is not 10calIy linear at its boundary points. In that
case the asymptotic nulI distribution of the likelihood ratio statistic is not ehi-square, but
the distribution of a certain functional of a Gaussian vector.
Besides for testing, the likelihood ratio statistic is often used for constructing confidenee
regions for a parameter ljJ (O). These are defined, as usual, as the values T for which a null
hypothesis Ho : ljJ (()) == T is not rejected. Asymptotic confidenee sets obtained by using the
ehi -square approximation are thought to be of better coverage accuracy than those obtained
by other asymptotic methods.
The likelihood ratio test has the desirable property of automatically achieving reduction
of the data by sufficiency: The test statistic depends on a minimal sufficient statistic only.
This is immediate from its definition as a quotient and the characterization of sufficiency by
the factorization theorem. Another property of the test is also immediate: The likelihood
ratio statistic is invariant under transformations of the parameter space that leave the nun
and alternative hypotheses invariant. This requirement is often imposed on test statistics
but is not necessarily desirable.
16.1 Example (Multinomiai vector). A vector N == (NI, .. . , Nk) thatpossesses themul-
tinomial distribution with parameters n and p == (Pl, . . . , Pk) can be viewed as the sum of
n independent multinomial vectors with parameters 1 and p. By the sufficiency reduction,
the likelihood ratio statistic based on N is the same as the statistic based on the single
observations. Thus our asymptotic results apply to the likelihood ratio statistic based on N,
if n ---+ 00.
lf the success probabilities are completely unknown, then their maximum likelihood
estimator is N/n. Thus, the log likelihood ratio statistic for testing a nulI hypothesis Ho : P E
Po against the alternative HI : P Po is given by
( n ) (Nl / n)NJ . . . (Nk / n)N k k ( N. )
NI...N k 2 . f " 1 1
210g N N == In Ni og - .
su p 'T) ( n ) P J . . . P k pEP o '- I npi
pEro NJ ...N k 1 k 1-
The fuU parameter set can be identified with an open subset of IRk-l , if P with zero coordi-
nates are excluded. The nulI hypothesis may take many forms. For a simple nun hypothesis
16.2 Taylor Expansion
229
the statistic is asymptotically ehi-square distributed with k - 1 degrees of freedom. This
follows from the general results in this chapter. t
Multinomial variables arise, among others, in testing goodness-of-fit. Suppose we wish
to test that the true distribution of a sample of size n belongs to a certain parametric model
{P() : e E 8}. Given a p arti ti on of the sample space into sets XI, . . . , X k , define NI, . . . , Nk
as the numbers of observations falling into each of the sets of the partition. Then the
vector N == (NI, . . . , Nk) possesses a multinomial distribution, and the original problem
can be translated in testing the null hypothesis that the success probabilities p have the
form (p()(X I ), ... , p()(X k )) for some e. O
16.2 Example (Exponentialfamilies). Suppose that the observations are sampled from
a density p() in the k-dimensional exponential family
p()(x) == c(e)h(x)e()Tt(x).
Let 8 C ]Rk be the natural parameter space, and consider testing a null hypothesis 8 0 c 8
versus its complement 8 - 80. The log likelihood ratio statistic is given by
An == 2n sup inf [(e - eO)Tt n + log c(e) - log c(e o ) J.
eE8 eE8 0
This is closely related to the Kullback - Leibler divergence of the measures p()o and Pe, which
is equal to
P() T
K (e, e o ) == P() log - == (e - e o ) p()t + log c(e) - log c(e o ).
p()o
If the maximum likelihood estimator e exists and is contained in the interior of 8, which
is the case with probability tending to 1 if the true parameter is contained in 8, then e
is the moment estimator that solves the equation P() t == t n. Comparing the two preceding
displays, we see that the likelihood ratio statistic can be written as An == 2n K (e, ( 0 ),
where K (e, ( 0 ) is the infimum of K (e, e o ) over e o E 8 0 . This pretty formula can be used
to study the asymptotic properties of the likelihood ratio statistic directly. Altematively, the
general results obtained in this chapter are applicable to exponential families. O
*16.2 Taylor Expansion
Write en,o and en for the maximum likelihood estimators for e if the parameter set is taken
equal to 8 0 or 8, respectively, and set l() == log p(). In this section assume that the true value
of the parameter 1J is an inner point of 8. The likelihood ratio statistic can be rewritten as
n
A == -2" ( l ( X. ) - f> ( X' )) ,
n .L-t ()n.O l ()n l
i=I
To find the limit behavior of this sequence of random variables, we might replace L l() (Xi)
by its Taylor expansion around the maximum likelihood estimator e == en. If e 1---+ l() (x)
t It is also proved in Chapter 17 by relating the likelihood ratio statistic to the ehi-square statistie.
230
Likelihood Ratio Tests
is twice continuously differentiable for every x, then there exists a vector en between 8n.o
and en such that the preceding display is equal to
n
-2(e n ,0 - en) L iOn (Xi) - (en,o - en)T L len (xž)(en,o - en).
i=l
Because en is the maximum likelihood estimator in the unrestrained model, the linear term in
this expansion vanishes as soon as en is an inner point of 8. If the averages - n -1 L f' e (X i)
converge in probability to the Fisher information matrix Ii} and the sequence -Jli(en,o - en)
is bounded in probability, then we obtain the approximation
An == -Jli(e n - en,O)T Ii}-Jli(e n - en,o) + 0PtJ (1).
(16.3)
In view of the results of Chapter 5, the latter conditions are reasonable if {} E 8 0 , for then
both en and en,o can be expected to be -Jli-consistent. The preceding approximation, if it
can be justified, sheds some light on the quality of the likelihood ratio test. It shows that,
asymptotically, the likelihood ratio test measures a certain distance between the maximum
likelihood estimators under the nulI and the full hypotheses. Such a procedure is intuitively
reasonable, even though many other distance measures could be used as well. The use of
the likelihood ratio statistic entails a choice as to how to weigh the different "directions" in
which the estimators may differ, and thus a choice of weights for "distributing power" over
different deviations. This is further studied in section 16.4.
If the nulI hypothesis is asingle point 8 0 == {eo}, then en,o == eo, and the quadratic form
in the preceding display reduces under Ho : e == e o (i.e., {} == e o ) to hnli}h n for h n == -Jli(e n -
f}) T. In view of the resuits of Chapter 5, the sequence h n can be expected to converge in
distribution to a variable h with a normal N (O, I;- 1 ) -distribution. Then the sequence An
converges under the nulI hypothesis in distribution to the quadratic form h T Ii} h. This is the
squared length of the standard normal vec tor I /2h, and possesses a ehi-square distribution
with k degrees of freedom. Thus the ehi -square approximation announced in the introduction
follows.
The situation is more complicated if the nulI hypothesis is composite. If the sequence
-Jli(en,o - f}, en - f}) converges jointly to a variable (ha, h), then the sequence An is
asymptotically distributed as (h - hO)T Ii) (h - ha). A nulI hypothesis 80 that is (a seg-
ment of) a lower dimensional affine linear subspace is itself a "regular" parametric model.
If it contains f} as arelative inner point, then the maximum likelihood estimator en,o
may be expected to be asymptotically normal within this affine subspace, and the pair
-Jli(en,o - fJ, en - fJ) may be expected to be jointly asymptotically norma!. Then the like-
lihood ratio statistic is asymptotically distributed as a quadratic form in normal variables.
Closer inspection shows that this quadratic form possesses a ehi-square distribution with
k - I degrees of freedom, where k and I are the dimensions of the fuU and null hypothe-
ses. In comparison with the case of a simple null hypothesis, I degrees of freedom are
"lost."
Because we shall rigorously derive the limit distribution by a different approach in
the next section, we make this argument preci se only in the particular case that the nulI
hypothesis 8 0 consists of all points (el, . . . , el, O, . . . , O), if e ranges over an open subset
8 of }Rk. Then the score function for e under the nulI hypothesis consists of the first I
coordinates of the score function f i} for the whole model, and the information matrix under
the null hypothesis is equal to the (I x I) prineipal submatrix of I ff. Write these as f i}, l and
Ii}, l, l, respectively, and use a similar partitioning notation for other vectors and matrices.
16.3 Using Local Asymptotic Normality
231
Under regularity conditions we have the linear approximations (see Theorem 5.39)
1 -1 .
,Jn (e n,O,SI - TJ Sl) = ,Jn b Iff,SI,SC i ff, Sl (Xi) + o P, (1),
1 -1 .
,Jn(()n - tJ) == ,Jn It} ft}(Xi) + oPđ(l).
n. 1
l=
Given these approximations, the multivariate central limit theorem and Slutsky's lemma
yield the joint asymptotie normality of the maximum likelihood estimators. From the form
of the asymptotic covarianee matrix we see, after some matrix manipulation,
,Jn(en, Z - en,O, Z) == -I;' z, zItJ, z,>z,Jn en,>z + op(l).
(Alternatively, this approximation follows from a Taylor expansion of O L7=1 ien, z
around en,o, z.) Substituting this in (16.3) and again carrying out some matrix manipulations,
we find that the likelihood ratio statistic is asymptotically equivalent to (see problem 16.5)
,Jn e:>l ((I; I tl,>l) -1,Jn en,>l.
(16.4)
The matrix (I;I»Z,>Z is the asymptotic covariance matrix of the sequence,Jn en,>z, whence
we obtain an asymptotic ehi-square distribution with k - I degrees of freedom, by the same
argument as before.
We close this section by relating the likelihood ratio statistic to two other test statistics.
U nder the simple null hypothesis 8 0 == {eo}, the likelihood ratio statistic is asymptotically
equivalent to both the maximum likelihood statistic (or Wald statistic) and the score statistic.
These are given by
n(e n - eo)T Ieo(e n - eo) and [t £eo(X i ) r Ig;;1 [t £eo(X i ) 1
The Wald statistic is a natural statistic, but it is often criticized for necessarily yielding
ellipsoidal confidence sets, even if the data are not symmetric. The score statistic has the
advantage that calculation of the supremum of the likelihood is unnecessary, but it appears
to perform less well for smaller values of n.
In the ease of a composite hypothesis, a Wald statistic is given in (16.4) and a score
statistic can be obtained by substituting the approximation ne n , >z (I; 1 ) >z, >z L i en o >1 (Xi)
in (16.4). (This approximation is obtainable from linearizing L(i en -ien,o)') In both cases
we also replace the unknown parameter i} by an estimator.
16.3 Using Local Asymptotic Normality
An insightful derivation of the asymptotic distribution of the likelihood ratio statistic is
based on convergence of experiments. This approach is possible for general experiments,
but this section is restricted to the case of local asymptotic normality. The approach applies
also in the case that the (1ocal) parameter spaces are not linear.
Introducing the local parameter spaces Hn == -Jfi(8 - tJ) and Hn,o == -Jfi(8 0 - 6), we
can write the likelihood ratio statistic in the form
A - 2 I TI7=lPtJ+h/yr,1(Xi) - 2 I TI7=lPfJ+h/yr,1(Xi)
n - sup og TI n sup og TI n .
hEHn i=lPfJ(X i ) hEHn,O i=lPfJ(X i )
232
Likelihood Ratio Tests
In Chapter 7 it is seen that, for large n, the rescaled likelihood ratio process in this display
is similar to the likelihood ratio proces s of the normal experiment (N (h, I; 1 ) : h E JR k) .
This suggests that, if the sets Hn and Hn,o converge in a suitable sense to sets H and Ha, the
sequence An converges in distribution to the random variable A obtained by substituting
the normallikelihood ratios, given by
dN(h,I;I) dN(h,I;I)
A == 2 sup log ( -1 ) (X) - 2 sup log ( -1 ) (X).
hEH dN O, If} hEHo dN O, If}
This is exactly the likelihood ratio statistic for testing the nuH hypothesis Ha : h E Ho versus
the alternative HI : h E H - Ho based on the observation X in the normal experiment.
Because the latter experiment is simple, this heuristic is useful not only to derive the
asymptotic distribution of the sequence An, but also to understand the asymptotic quality
of the corresponding sequence of tests.
The likelihood ratio statistic for the normal experiment is
A == inf (X - h)T If}(X - h) - inf(X - h)T If}(X - h)
hEHo hEH
== III /2X - I /2HoI12 -III /2X _ I /2HI12.
( 16.5)
The distribution of the sequence An under f} corresponds to the distribution of A under
h == O. Under h == O the vec tor I /2 X possesses a standard normal distribution. The foHowing
lemma shows that the squared distance of a standard normal variable to a linear subspace
is chi square-distributed and hence explains the chi -square limit when Ho is a linear space.
16.6 Lemma. Let X be a k-dimensional random veetor with a standard normal distri-
bution and let Ho be an l-dimensional linear subspaee of IR. k . Then II X - Ho 11 2 is ehi
square-distributed with k - l degrees of freedom.
Proof. Take an orthonormal base ofJRk such that the first l elements span Ho. By Pythago-
ras' theorem, the squared distance of a vector z to the space Ho equals the sum of squares
Li>l zf of its last k -l coordinates with respect to this basis. A change of base corresponds
to an orthogonal transformation of the coordinates. Because the standard normal distribu-
tion is invariant under orthogonal transformations, the coordinates of X with respect to any
orthonormal base are independent standard normal variables. Thus II X - Ho 11 2 == Li>l Xl
is chi square-distributed. .
If f} is an inner point of 8, then the set H is the fuH space JRk and the second term on
the right of (16.5) is zero. Thus, if the local nuH parameter spaces -Jfi(8 0 - f}) converge
to a linear subspace of dimension I, then the asymptotic nuH distribution of the likelihood
ratio statistic is chi -square with k - l degrees of freedom.
The following theorem makes the preceding informal derivation rigorous under the same
mild conditions employed to obtain the asymptotic normality of the maximum likelihood
estimator in Chapter 5. It uses the foHowing notion of eonvergenee of sets. Write Hn ---+ H
if H is the set of all limit s lim h n of converging sequences h n with h n E Hn for every n
and, moreover, the limit h == limi hn; of every converging sequence h ni with h ni E H ni for
every i is contained in H.
16.3 Using Local Asymptotic Normality
233
16.7 Theorem. Let the model (Pe : 8 E 8) be differentiable in quadratic me an at il with
nonsingular Fisher information matrix, and suppose that for every el and 8 2 in a neighbor-
. .2
hood of il and for a measurable function -e such that P ff -e < 00,
Ilog Pe l (x) - log P e 2 (x) I < f(x) 118 1 - 8 2 11.
If the maximum likelihood estimators e n,a and en are consistent under il and the sets Hn,a
and Hn converge to sets Ha and H, then the sequence of likelihood ratio statistics An
converges under il + h / ,Jli in distribution to A given in (16.5), for X normally distributed
with mean h and covariance matrix Ii 1.
* Pro of. Let CG n == ,Jli (P n - p ff) be the empirical process, and define stochastic processes
Zn by
Pff+h/,;n T . 1 T
Zn(h) == nPn log - h CGn-e ff + -h Iffh.
Pff 2
The differentiability of the model implies that Zn (h) O for every h. In the proof of
Theorem 7.12 this is strengthened to the uniform convergence
sup IZn(h)1 O,
IIhll::sM
Furthermore, it follows from this proof that both en,a and en are ,Jli-consistent under il.
(These statements can also be proved by elementary arguments, but under stronger regularity
conditions.)
The preceding display is also valid for every sequence M n that increases to 00 suffi-
ciently slowly. Fix such a sequence. By the ,Jli-consistency, the estimators e n,a and en are
contained in the ball of radius Mn/,Jli around il with probability tending to 1. Thus, the
limit distribution of An does not change if we replace the sets Hn and Hn,a in its definition
by the sets Hn n ball (O, M n ) and Hn,a n ball (O, M n ). These "truncated" sequences of sets
stiH converge to H and Ha, respectively. Now, by the uniform convergence to zero of the
processes Zn (h) on Hn and Hn,a, and simple algebra,
every M.
Pff+h/,;n Pff+h/,;n
An == 2 sup nP n log - 2 sup nP n log
hEHn Pff hEHn,o Pff
( T . 1 T ) ( T' 1 T )
== 2 sup h CGn£ff - 2h Iffh - 2 sup h CGn£ff - 2h Iffh + op(l)
hEHn hEHn,o
== II I;I/2 CGn i ff - I /2 Ha 11 2 - II I;I/2 CGn i ff - I /2 H 11 2 + o p (1)
by Lemma 7.13 (ii) and (iii). The theorem foHows by the continuous-mapping theorem. .
16.8 Example (Generalized linear models). In a generalized linear model a typical ob-
servation (X, Y), consisting of a "covariate vector" X and a "response" Y, possesses a
density of the form
P{3(x, y) == e yk ({3T x )cp-bok({3T x )cpccp(y)px(x).
(It may be more natural to model the covariates as (observed) constants, but to fit the model
into our i.i.d. setup, we consider them to be arandom sample from a density p x.) Thus, given
234
Likelihood Ratio Tests
X, the variable Y follows an exponential family density eyeq;-bce)q; Ccp (y) with parameters e ==
k(f3T X) and ej;. Using the identities for exponential families based on Lemma 4.5, we obtain
Et3 (Y I X) == bI o k(f3T X),
bIlo k(f3T X)
var ,8,<1> (Y I X) = eP '
The function (bIo k)-I is called the linkfunction of the model and is assumed known. To
make the parameter f3 identifiable, we assume that the matrix EX X T exists and is nonsin-
gular.
To judge the goodness-of-fit of a generalized linear model to a given set of data (XI,
YI), . . . , (X n , Y n ), it is customary to calculate, for fixed ej;, the log likelihood ratio statistic
for testing the model as described previously within the model in which each Yi, given Xi,
stiH follows the given exponential family density, but in which the parameters e (and hence
the conditional means E( Yi 1 Xi)) are aHowed to be arbitrary values e i , unrelated across
the n observations (Xi, Yi). This statistic, with the parameter ej; set to 1, is known as the
deviance, and takes the form, with n the maximum likelihood estimator for f3, t
sup TI e Yik (t3 T Xi)-b o k(t3 T Xd
D(Y n , (l) = -21og 1=1 TIn eYiĐi-bCĐi)
Pe] ,...,e n l=1
n
= -2 L[ Yi (k( Xi) - (b')-l(y;)) - b O k( Xi) + b O (b')-l (Yi) l
i=1
In our present setup, the codimension of the nulI hypothesis within the "fuH model" is
equal to n - k, if f3 is k -dimensional, and hence the preceding theory does not appl y
to the deviance. (This could be different if there are multiple responses for every given
covariate and the asymptotics are relative to the number of responses.) On the other hand,
the preceding theory allows an "analysis of deviance" to test nested sequences of regres sion
models corresponding to inclusion or exclusion of a given covariate (i.e., column of the
regression matrix). For instance, if Di (Y n, {lU)) is the deviance of the model in which
the i + 1, i + 2, . . . , kth coordinates of f3 are a priori set to zero, then the difference
Di-I(Y n , {lu-I)) - Di(Y n , {lei)) is the log likelihood ratio statistic for testing that the ith
coordinate of f3 is zero within the model in which all higher coordinates are zero. According
to the theory of this chapter, eP times this statistic is asymptoticaHy ehi square-distributed
with one degree of freedom under the smaller of the two models.
To see this formally, it suffices to verify the conditions of the preceding theorem. Using
the identities for exponential families based on Lemrna 4.5, the score function and Fisher
information matrix can be computed to be
f t3 (x, y) == (y - b' o k(f3T x))k l (f3T x)x,
[13 == Eb ll o k(fJT X)k l ({JT X)2 X XY.
Depending on the function k, these are very well-behaved functions of (J, because b is
a strictly convex, analytic function on the interior of the natural parameter space of the
family, as is seen in section 4.2. Under reasonable conditions the function SUPt3EU II f 1311 is
t The arguments Y n and il of D are the veetors of estimated (eonditional) means of Y given the full model and
the generalized linear model, respeetively. Thus ili = b' o k( Xi).
16.3 Using Local Asymptotic Normality
235
square-integrable, for every small neighborhood U, and the Fisher information is continu-
ous. Thus, the local conditions on the model are easily satisfied.
Proving the consistency of the maximum likelihood estimator may be more involved,
depending on the link function. If the parameter f3 is restricted to a compact set, then most
approaches to proving consistency apply without further work, including Wald' s method,
Theorem 5.7, and the classical approach of section 5.7. The last is particularly attractive
in the case of canonicallink functions, which correspond to setting k equal to the identity.
Then the second -derivative matrix l f3 is equal to - b" (f3 T x)x X T, whence the likelihood
is a strictly concave function of f3 whenever the observed covariate vectors are of full
ranko Consequently, the point of maximum of the likelihood function is unique and hence
consistent under the conditions of Theorem 5.14. t O
16.9 Example (Location scale). Suppose we observe a sample from the density f ((x -
J-l) / o' ) / o' for a given probability density f, and a location-scale parameter e == (J-l, 0')
ranging over the set 8 == x +. We consider two testing problems.
(i). Testing Ho : J-l == O versus HI : J-l =I- O corresponds to setting 8 0 == {O} X IR+. For a
given point == (O, 0') from thenullhypothesis the set-Jn(8 0 - ) equals {O}x (--Jna, (0)
and converges to the line ar space {O} x . Under regularity conditions on f, the sequence
of likelihood ratio statistics is asymptotically chi square-distributed with 1 degree of
freedom.
(ii). Testing Ho : J-l < O versus HI : J-l > O corresponds to setting 8 0 == (-00, O] X IR+.
For a given point == (O, 0') on the boundary of the null hypothesis, the sets -Jn(8 0 -
) converge to Ho == (-00, O] x . In this case, the limit distribution of the likelihood
ratio statistics is not chi -square but equals the distribution of the square distance of a
standard normal vector to the set I /2 Ho == {h : (h, I;; 1 /2 el) < O}. The latter is a half -space
with boundary line through the origin. Because a standard normal vector is rotationally
symmetric, the distribution of its distance to a half-space of this type does not depend on the
orientation of the half-space. Thus the limit distribution is equal to the distribution of the
squared distance of a standard normal vector to the half-space {h : h 2 < O}: the distribution
of (Z v 0)2 for a standard normal variable Z. Because P( (Z v 0)2 > c) == p(Z2 > c) for
every c > O, we must choose the critical value of the test equal to the upper 2a -quantile of
the chi -square distribution with 1 degree of freedom. Then the asymptotic level of the test
is a for every tJ on the boundary of the nun hypothesis (provided a < 1/2).
For a point in the interior of the null hypothesis Ho : J-l < O the sets -Jn (8 0 - )
converge to x IR and the sequence of likelihood ratio statistics converges in distribution to
the squared distance to the whole space, which is zero. This means that the probability of an
error of the first kind converges to zero for every tJ in the interior of the null hypothesis. O
16.10 Example (Testing a ball). Suppose we wish to test the null hypothesis Ho : II ()" < 1
that the parameter belongs to the unit ball versus the alternative HI : "e" > 1 that this is
not case.
If the true parameter f} belongs to the interior of the null hypothesis, then the sets
-Jn (8 0 - tJ) converge to the whole space, whence the sequence of likelihood ratio statistics
converges in distribution to zero.
t For a detailed study of suffieient eonditions for eonsisteney see [45].
236
Likelihood Ratio Tests
For f} on the boundary of the unit ball, the sets vn(8 0 - f}) grow to the half-space
Ho == { h : (h, f}) < O}. The sequence of likelihood ratio statistics converges in distribution
to the distribution of the square distance of a standard normal vector to the half-space
I /2 Ho == {h: (h, I;1/2f}) < O}. By the same argument as in the preceding example, this
is the distribution of (Z v 0)2 for a standard normal variable Z. Once again we find an
asymptotic level-a test by using a 2a -quantile. D
16.11 Example (Testing a range). Suppose that the nulI hypothesis is equal to the image
8 0 == g(T) of an open subset T of a Euclidean space of dimension I < k. If g is a
homeomorphism, continuously differentiable, and of full rank, then the sets vn ( 8 0 - g (T) )
converge to the range of the derivative of g at T, which is a subspace of dimension I.
lndeed, for any 17 E JRz the vectors T + TJ I vn are contained in T for sufficiently large
n, and the sequence vn (g (T + 17 I vn) - g ( T )) converges to g 17 . Furthermore, if a sub-
sequenee of vn(g(t n ) - g('r)) eonverges to a point h for a given sequence t n in T, then
the corresponding subsequence of vn(t n - T) converges to 17 == (g-l ) (T:)h by the differ-
entiability of the inverse mapping g-l and hence vn(g(t n ) - g(T)) g 17. (We can use
the rank theorem to give a precise definition of the differentiability of the map g -} on the
manifold g(T).) D
16.4 Asymptotic Power Functions
Because the sequence of likelihood ratio statistics converges to the likelihood ratio statistic
in the Gaussian limit experiment, the likelihood ratio test is "asymptotically effieient" in
the same way as the likelihood ratio statistic in the limit experiment is "efficient." If
the local limit parameter set Ho is a half-space or a hyperplane, then the latter test is
uniformly most powerful, and hence the likelihood ratio tests are asymptotieally optimal
(see Proposition 15.2).This is the case, in p articul ar, for testing a simple null hypothesis
in a one-dimensional parametrie model. On the other hand, if the hypotheses are higher-
dimensional, then there is often no single best test, not even under reasonable restrictions on
the class of admitted tests. For different (one-dimensional) deviations of the null hypothesis,
different tests are optimal (see the discussion in Chapter 15). The likelihood ratio test is an
omnibus test that gives reasonable power in all directions. In this section we study its local
asymptotic power function more elosely.
We assume that the parameter f} is an inner point of the parameter set and denote the true
parameter by f} + hi vn. Under the conditions of Theorem 16.7, the sequence oflikelihood
ratio statistics is asymptotically distributed as
A == II Z + I /2h - I /2 Ho 11 2
for a standard normal vec tor Z. Suppose that the limiting loeal parameter set Ho is a linear
subspace of dimension I, and that the nulI hypothesis is rejected for values of An exeeeding
the eritical value xl-z ,ex' Then the local power functions of the resulting tests satisfy
Trn (1'} + ) = PHhlv'n(An > xL.cJ --* Ph(A > XL,a) =: Tr(h).
The variable A is the squared distance of the vector Z to the affine subspace - I /2 h + I /2 Ho.
By the rotational invariance of the normal distribution, the distribution of A doe s not de-
pend on the orientation of the affine subspace, but only on its codimension and its distance
16.4 Asymptotic Power Functions
237
8 == II I j2h - I j2 Hall to the origin. This distribution is known as the noncentral ehi-square
distribution with noneentrality parameter 8. Thus
n(h) = P( xL (II I /2h - I /2 Ha II) > xL,a).
The noneentral ehi-square distributions are stoehastieally inereasing in the noneentrality
parameter. It follows that the likelihood ratio test has good (loeal) power at h that yield a
large value of the noneentrality parameter.
The shape of the asymptotie power funetion is easiest to understand in the ease of a
simple null hypothesis. Then Ha == {O}, and the noneentrality parameter reduees to the
square root of h T Ifj h. For h == JLhe equal to a multiple of an eigenveetor he C of unit norm)
of Ifj with eigenvalue Ae, the noneentrality parameter equals -AJL. The asymptotie power
funetion in the direetion of he equals
n(JLh e ) == P(Xl( eJL) > Xl,a).
The test performs best for departures from the null hypothesis in the direetion of the eigen-
veetor eorresponding to the largest eigenvalue. Even though the likelihood ratio test gives
power in all direetions, it does not treat the direetions equally. This may be worrisome if
the eigenvalues are very inhomogeneous.
Further insight is gained by eomparing the likelihood ratio test to tests that are designed
to be optimal in given direetions. Let X be an observation in the limit experiment, having a
N(h, I;l)-distribution. The test that rejeets the null hypothesis Ha == {O} if I e h; XI >
Zaj2 has level a and power funetion
1fhe (JLhe) == p(xlc eJL) > Xl,a).
For large k this is a eonsiderably better power funetion than the power funetion of the
likelihood ratio test (Figure 16.1), but the forms of the power funetions are similar. In
particular, the optimal power functions show a similar dependenee on the eigenvalues of
o
T"""
o
o
o
C\J
o
o
5
20
25
10
15
Figure 16.1. The functions /1-2 P(Xf(J1) > Xf,a) for k == 1 (solid), k == 5 (dotted) and k == 15
(dashed), respectively, for a == 0.05.
238
Likelihood Ratio Tests
the eovarianee matrix. In this sense, the apparently unequal distribution of power over the
different direetions is not unfair in that it refleets the intrinsie difficulty of deteeting ehanges
in different direetions. This is not to say that we should never ehange the (automatie)
emphasis given by the likelihood ratio test.
16.5 Bartlett Correction
The ehi -square approximation to the distribution of the likelihood ratio statistie is relativel y
aeeurate but can be mueh improved by a eorreetion. This was first noted in the example
of testing for inequality of the varianees in the one-way layout by Bartlett and has sinee
been generalized. Although every approximation can be improved, the Bartlett correction
appears to enjoy a partieular popularity.
The eorreetion takes the form of a eorreetion of the (asymptotie) mean of the likelihood
ratio statistie. In regular eases the distribution of the likelihood ratio statistie is asymptoti-
eally ehi-square with, say, r degrees offreedom, whenee its mean ought to be approximately
equal to r. Bartlett's eorreetion is intended to make the mean exaetly equal to r, by replacing
the likelihood ratio statistie An by
rAn
EeoAn
The distribution of this statistie is next approximated by a ehi -square distribution with r
degrees of freedom. Unfortunately, the mean EeoAn may be hard to ealeulate, and may
depend on an unknown null parameter e o . Therefore, one first obtains an expression for the
mean of the form
b ( ()o )
Eeo An == 1 + + . . . .
n
Next, with b n an appropriate estimator for the parameter b(eo), the eorreeted statistic takes
the form
rAn
1 + bn/n
The surprising faet is that this reeipe works in some generality. Ordinarily, improved approx-
imations would be obtained by writing down and next inverting an Edgeworth expansion
of the probabilities P(An < x); the eorreetion would depend on x. In the present ease this
is equivaIent to a simple eorreetion of the mean, independent of x. The teehnieal reason is
that the polynomial in x in the (l/n)-term of the Edgeworth expansion is of degree 1. t
*16.6 Bahadur Efficiency
The eIaim in the Seetion 16.4 that in many situations "asymptotieally optimal" tests do not
exist refers to the study of effieieney relative to the Ioeal Gaussian approximations described
t For a further diseussion, see [5], [9], and [83], and the referenees eited there.
16. 6 Bahadur Efficiency
239
in Chapter 7. The purpose of this section is to show that, under regularity conditions, the
likelihood ratio test is asymptoticalIy optimal in a different setting, the one of Bahadur
efficiency.
For simplicity we restrict ourselves to the testing of finite hypotheses. Given finite sets
Po and Pl of probability measures on a measurable space (X, A) and arandom sample
XI, . . . , X n , we study the log likelihood ratio statistic
A - - I SUPQEPl TI7=lq(X i )
n - og TI n .
SUPPEPo i=IP(X i )
More general hypotheses can be treated, underregularity conditions, by finite approximation
(see e.g., Section 10 of [4]).
The observed level of a test that rejects for large values of a statistic Tn is defined as
L n == sup P p (Tn > t) I t= Tn .
PEPo
The test that rejects the nun hypothesis if Ln < ex has level ex. The power of this test is
maximal if Ln is "minimaI" under the alternative (in a stochastic sense). The Bahadur
slope under the alternative Q is defined as the limit in probability under Q (if it exists)
of the sequence (- 21 n) log Ln. If this is "large," then Ln is smalI and hence we prefer
sequences of test statistics that have a large slope. The same conclusion is reached in
section 14.4 by considering the asymptotic relative Bahadur efficiencies. It is indicated
there that the Neyman-Pearson tests for testing the simple nulI and alternative hypotheses
p and Q have Bahadur slope -2Q log(plq). Because these are the most powerful tests,
this is the maximal slope for testing P versus Q. (We give a precise proof in the folIowing
theorem.) Consequently, the slope for a general nun hypothesis cannot be bigger than
inf PEPo -2 Q log(p 1 q). The sequence of likelihood ratio statistics attains equality, even if
the alternative hypothesis is composite.
16.12 Theorem. The Bahadur slope of any sequence of test statistics for testing an arbi-
trary nul! hypothesis Ha : P E Po versus a simple alternative HI : P == Q is bounded above
by infPEPo -2Q log(plq), for any probability measure Q. IfPa and Pl are finite sets of
probability measures, then the sequence of likelihood ratio statistics for testing Ha : P E Po
versus HI : P E Pl attains equality for every Q E Pl.
Proof. Because the observed level is a supremum over Po, it suffices to prove the upper
bound of the theorem for a simple nulI hypothesis Po == {P}. If -2Q log(plq) == 00,
then there is nothing to prave. Thus, we can assume without loss of generality that Q is
absolutely continuous with respect to P. Write An for log TI 7= I (q 1 P )(X i ). Then, for any
constants B > A > Qlog(qlp),
PQ(L n < e- nB , An < nA) == E p 1{L n < e- nB , An < nA}e An
< enAPp(Ln < e- nB ).
Because Ln is superuniformly distributed under the nulI hypothesis, the last expression
is bounded above by exp -n(B - A). Thus, the sum of the probabilities on the left side
over n E N is finite, whence - (21 n) log Ln < 2B or An > nA for all sufficiently large
n, almost surely under Q, by the Borel-Cantelli lemma. Because the sequence n-I An
240
Likelihood Ratio Tests
converges almost surely under Q to Q log(q / p) < A, by the strong law oflarge numbers, the
second possibility can occur only finitely many times. It folIows that -(2/n) log Ln < 2B
eventually, almost surely under Q. This having been established for any B > Q log(q / p),
the pro of of the first assertion is complete.
To prove that the likelihood ratio statistic attains equality, it suffices to prove that its slope
is bigger than the upper bound. Write An for the log likelihood ratio statistic, and write
sup p and sup Q for suprema over the nulI and alternative hypotheses. Because (1/ n) An is
bounded above by sUPQ IFn log(q / p), we have, by Markov's inequality,
Pp( An > t) < Pp(JP'nlOg ; > t) < IPllmgxe-ntEpenll'n[Og(qjP).
The expectation on the right side is the nth power of the integral f (q / p) d P == Q (p > O) <
1. Take logarithms left and right and multiply with - (2/ n) to find that
2 ( 1 - ) 2log IPII
- - log P p - An > t > 2t - .
n n n
Because this is valid uniformly in t and P, we can take the infimum over P on the left
side; next evaluate the left and right sides at t == (1/ n) An. By the law of large numbers,
IF n log(q / p) --+ Q log(q / p) almost surely under Q, and this remains valid if we first add
the infimum over the (finite) set Po on both sides. Thus, the limit inferior of the sequence
(1/ n) An > inf p IF n log (q / p) is bounded below by inf p Q log (q / p) almost surel y under
Q, where we interpret Q log(q / p) as 00 if Q(p == O) > O. Insert this lower bound in
the preceding display to conclude that the Bahadur slope of the likelihood ratio statistics is
bounded below by 2 inf p Q log(q / p). .
Notes
The classical reference s on the asymptotic nulI distribution of likelihood ratio statistic are
papers by Chernoff [21] and Wilks [150]. Our main theorem appears to be better than
Chernoff's, who us es the "classical regularity conditions" and a different notion of approx-
imation of sets, but is not essentialIy different. Wilks' treatment would not be acceptable to
present -day referees but maybe is not so different either. He appears to be saying that we
can replace the originallikelihood by the likelihood for having observed only the maximum
likelihood estimator (the error is asymptoticalIy negligible), next refers to work by Doob
to infer that this is a Gaussian likelihood, and continues to compute the likelihood ratio
statistic for a Gaussian likelihood, which is easy, as we have seen. The approach using a
Taylor expansion and the asymptotic distributions of both likelihood estimators is one way
to make the argument rigorous, but it seems to hide the original intuition.
Bahadur [3] presented the efficiency of the likelihood ratio statistic at the fifth Berkeley
symposium. Kallenberg [84] shows that the likelihood ratio statistic remains asymptotically
optimal in the setting in which both the desired level and the alternative tend to zero, at least
in exponential families. As the pro of of Theorem 16.12 shows, the composite nature of the
alternative hypothesis "disappears" elegantly by taking (1/ n) log of the error probabilities -
too elegantly to attach much value to this type of optimality?
Problems
241
PROBLEMS
1. Let (XI, YI),..., (X n , Y n ) be a sample from the bivariate normal distribution with mean vec-
tor (IL, v) and covariance matrix the diagonal matrix with entries (J2 and i 2 . Calculate (or
characterize) the likelihood ratio statistic for testing Ha : IL == v versus HI : J.L -=I v.
2. Let N be a kr-dimensional multinomial variable written as a (k x r) matrix (Ni}). Calculate
the likelihood ratio statistic for testing the nulI hypothesis of independence Ha : Pij == Pi. p.j for
every i and j. Here the dot denotes summation over all columns and rows, respectively. What is
the limit distribution under the null hypothesis?
3. Calculate the likelihood ratio statistic for testing Ha : IL == v based on independent samples of
size n from multivariate normal distributions N r (IL, ) and N r (v, ). The matrix is unknown.
What is the limit distribution under the null hypothesis?
4. Calculate the likelihood ratio statistic for testing Ha : IL 1 == ... == ILk based on k independent
samples of size n from N (IL j, (J2)-distributions. What is the asymptotic distribution under the
nulI hypothesis?
5. Show that (L;I»Z,>Z is the inverse of the matrix IfJ,>z,>z - IfJ,>z, zI;' z, zIfJ, z,>z.
6. Study the asymptotic distribution of the sequence An if the true parameter is contained in both
the null and alternative hypotheses.
7. Study the asymptotic distribution of the likelihood ratio statistics for testing the hypothesis
Ha : (J == - i based on a sample of size n from the uniform distribution on [(J, i]. Does the
asymptotic distribution correspond to a likelihood ratio statistic in alimit experiment?
17
Chi-Square Tests
The ehi-square statistie for testing hypotheses eoneerning multinomial
distributions derives its name from the asymptotie approximation to its
distribution. Two important applieations are the testing of independenee
in a two-way elassifieation and the testing of goodness-of-fit. In the seeond
applieation the multinomial distribution is ereated artijieially by group-
ing the data, and the asymptotie ehi-square approximation may be lost if
the original data are used to estimate nuisanee parameters.
17.1 Quadratic Forms in Normal Vectors
The ehi-square distribution with k degrees of freedom is (by definition) the distribution
of L =l Zl for i.i.d. N (O, l)-distributed variables ZI, . . . , Zk. The sum of squares is the
squared norm II Z 11 2 of the standard normal vector Z == (ZI, . . . , Zk). The following lemma
gives a characterization of the distribution of the norm of a general zero- mean normal vector.
17.1 Lemma. Ifthe veetor X is Nk (O, 'b)-distributed, then IIXII 2 is distributed as L =lAi
Zl for i. i. d. N (O, 1) -distributed variables ZI, . . . , Zk and A 1, . . . , Ak the eigenvalues of 'b.
Proof. There exists an orthogonal matrix O such that O'b O T == diag (Ai)' Then the
vector O X is Nk (O, diag (Ai) )-distributed, which is the same as the distribution of the vector
( Z1, . . . , J}:kZk)' Now IIX 11 2 == 110 X 11 2 has the same distribution as L(,J Zi)2. .
The distribution of a quadratic form of the type L =l Ai Zl is complicated in general.
However, in the case that every Ai is either O or 1, it reduces to a ehi-square distribution. If
this is not naturally the case in an application, then a statistic is often transformed to aehieve
this desirable situation. The definition of the Pearson statistic illustrates this.
17.2 Pearson Statistic
Suppose that we observe a vec tor X n == (X n ,1, . . . , Xn,k) with the multinomial distribution
corresponding to n trials and k classes having probabilities p == (Pl, . . . , Pk). The Pearson
242
17.2 Pearson Statistic
243
statistic for testing the nulI hypothesis Ha : p == a is given by
k ( X . - na. ) 2
Cn(a) == L n,l l .
i=l nai
We shalI show that the sequence C n (a) converges in distribution to a ehi-square distribution
if the nulI hypothesis is true. The practical relevanee is that we can use the ehi-square table
to find critical values for the test. The proof shows why Pearson divided the squares by nai
and did not propose the simpier statistic IIX n - na11 2 .
17.2 Theorem. If the vectors X n are multinomially distributed with parameters n and
a == (al, . . . , ak) > O, then the sequence C n (a) converges under a in distribution to the
xl-l -distribution.
Proof. The vector X n can be thought of as the sum of n independent multinomial vectors
YI, . . . , Y n with parameters 1 and a == (al, . . . , ak). Then
al (1 - al) -ala2
-a2 a l a2(1 - a2)
-alak
-a2 a k
EY i == a,
Cov Yi ==
-akal
-ak a 2
ak(l - ak)
B Y the multivariate central limit theorem, the sequence n-I /2 (X n - na) converges in distribu -
tion to the Nk(O, Cov Yl)-distribution. Consequently, with -Ja the vector with coordinates
va;,
( Xnl-na l Xnk-nak ) T
'o.., N(O, I - -Ja.JQ ).
na 1 nak
Because L ai == 1, the matrix I - -Ja-JaT has eigenvalue O, of multiplicity 1 (with eigen-
space spanned by -Ja), and eigenvalue 1, of multiplicity (k - 1) (with eigenspace equal to
the orthocomplement of -Ja). An application of the continuous-mapping theorem and next
Lemma 17.1 conclude the proof. .
The number of degrees of freedom in the ehi -squared approximation for Pearson' s statis-
tic is the number of cells of the multinomial vec tor that have positive probability. However,
the quality of the approximation also depends on the size of the cell probabilities aj. For
instance, if 1001 cells have nulI probabilities 10- 23 , . . . , 10- 23 , 1 - 10- 20 , then it is clear
that for moderate values of n all cells except one are empty, and a huge value of n is
necessary to make a Xlooo-approximation work. As a rule of thumb, it is often advised to
choose the partitioning sets such that each number naj is at least 5. This criterion depends
on the (possibly unknown) null distribution and is not the same as saying that the number
of observations in each cell must satisfy an absolute lower bound, which could be very
unlikely if the null hypothesis is false. The rule of thumb means to protect the level.
The Pearson statistic is oddly asymmetric in the observed and the true frequencies (which
is motivated by the form of the asymptotic covariance matrix). One method to symmetrize
244
Chi-Square Tests
the statistic leads to the Hellinger statistie
2 (Xn,i - nai)2 rv- 2
Hn (a) = 4 L...., ;x;; = 4 L...., (v' Xn,i - ,fiUli) .
i=l ( Xn,i + ,jlUii)2 i=l
Up to a multiplicative constant this is the Hellinger distance between the discrete probabil-
ity distributions on {I, . . . , k} with probability vectors a and Xn/ n, respectively. Because
X n / n - a O, the Hellinger statistic is asymptotically equivalent to the Pearson statistic.
17.3 Estimated Parameters
Chi-square tests are used quite often, but usually to test more complicated hypotheses. If
the nuU hypothesis of interest is composite, then the parameter a is unknown and eannot
be used in the definition of a test statistic. A natural extension is to replaee the parameter
by an estimate an and use the statistie
k (X A ) 2
. - na .
C ( A ) _" n,l n,l
n an - A .
. 1 nan i
l= ,
The estimator an is eonstrueted to be a good estimator if the nulI hypothesis is true. The
asymptotic distribution of this modified Pearson statistie is not necessarily ehi-square but
depends on the estimators an being used. Most often the estimators are asymptotically
normal, and the statistics
Jnan,i
Xn,i - nan,i
Jnan,i
vn(an,i - an,i)
;a;:;
X n i - na n i
, ,
are asymptotieally normal as well. Then the modified ehi -square statistic is asymptotieally
distributed as a quadratie form in a multivariate-normal veetor. In general, the eigenvalues
determining this form are not restrieted to O or 1, and their values may depend on the
unknown parameter. Then the eritieal value eannot be taken from a table of the ehi-square
distribution. There are two popular possibilities to avoid this problem.
First, the Pearson statistie is a eertain quadratie form in the observations that is motivated
by the asymptotie eovarianee matrix of a multinomial veetor. If the parameter a is estimated,
the asymptotie eovarianee matrix ehanges in form, and it is natural to ehange the quadratie
form in such away that the resulting statistie is again ehi -square distributed. This idea leads
to the Rao-Robson-Nikulin modifieation of the Pearson statistic, of which we diseuss an
example in seetion 17.5.
Seeond, we ean retain the form of the Pearson statistie but use speeial estimators a. In
partieular, the maximum likelihood estimator based on the multinomial veetor X n, or the
minimum-ehi square estimator a n defined by, with Po being the nun hypothesis,
(X n . i - n a n ,i)2 _ . f (Xn,i - npi)2
-ln .
i=l n a n,i PEP o i=l npi
The right side of this display is the "minimum-ehi square distance" of the observed frequen-
cies to the nulI hypothesis and is an intuitively reasonable test statistie. The nulI hypothesis
17.3 Estimated Parameters
245
is rejeeted if the distance of the observed frequeney vector Xn/ n to the set Po is large. A
disadvantage is greater eomputational eomplexity.
These two modifieations, using the minimum-ehi square estimator or the maximum
likelihood estimator based on X n , may seem natural but are artificial in some applieations.
For instance, in goodness-of- fit testing, the multinomial vector is formed by grouping the
"raw data," and it is more natural to base the estimators on the raw data rather than on the
grouped data. On the other hand, using the maximum likelihood or minimum-ehi square
estimator based on X n has the advantage of a remarkably simple limit theory: If the null
hypothesis is "loeally 1inear," then the modified Pearson statistie is again asymptotieally
ehi-square distributed, but with the number of degrees of freedom redueed by the (Ioeal)
dimension of the estimated parameter.
This interesting asymptotie result is most easily explained in terms of the minimum-
ehi square statistie, as the loss of degrees of freedom eorresponds to a projeetion (i.e., a
minimum distance) of the limiting normal vector. We shall first show that the two types
of modifieations are asymptotieally equivalent and are asymptotieally equivalent to the
likelihood ratio statistie as well. The likelihood ratio statistie for testing the null hypothesis
Ha: p E Po is given by (see Example 16.1)
Ln(a n ) == inf Ln(p),
PEPo
k
"" Xn,i
Ln(P) == 2 Xn,i log-.
i=l npi
17.3 Lemma. Let Po be a closed subset of the unit simplex, and let an be the maximum
likelihood estimator of a under the nulI hypothesis Ha : a E Po (based on X n ). Then
k (X . _ np.)2
inf L n,l 1 == C n (an) + o p (1) == Ln (an) + o p (1).
PEPo i=l npi
Proof. Let a n be the minimum-ehi square estimator of a under the null hypothesis. Both
sequenees of estimators a n and an are -Jll-eonsistent. For the maximum likelihood esti-
mator this follows from Corollary 5.53. The minimum-ehi square estimator satisfies by its
definition
k (X - ) 2 k (X ) 2
. - na . . - na.
L n,l _ n,l < L n,l 1 == Op(l).
i=l nan,i i=l nai
This implies that eaeh term in the sum on the left is O p (1), whenee n l a n,i - ai 1 2
O p ( a n,i) + O p (IXn,i - nai 1 2 / n) and henee the -Jll-eonsisteney.
Next, the two-term Taylor expansion 10g(1 + x) == x - x2 + 0(x 2 ) eombined with
Lemma 2.12 yields, for any -Jll-eonsistent estimator sequenee Pn,
k
L Xni
Xn,i log /'.'
n P .
i=l n,l
k ( /'. ) 1 k ( /'. ) 2
== - ""X . npn,i _ 1 + - ""x . npn,i - 1 + o ( 1 )
. n.l X. 2 n,l X. P
i=l n,l i=l n,l
1 k ( X . -n p /'. . ) 2
== 0+ - L n,l n,l + op(l).
2 i=l Xn,i
In the last expression we can also replaee Xn,i in the denominator by n Pn,i, so that we find
the relation Ln (Pn) == C n (Pn) between the likelihood ratio and the Pearson statistic, for
246
Chi-Square Tests
every ,Jn-consistent estimator sequence Pn. By the definitions of a n and an, we conclude
that, up to o p (1)-terms, C n ( an ) < C n (an) == Ln (an) < Ln ( an ) == C n ( a n). The lemma
follows. .
The asymptotic behavior of likelihood ratio statistics is diseussed in general in Chap-
ter 16. In view of the preceding lemma, we can now refer to this chapter to obtain the asymp-
totie distribution of the ehi-square statisties. Alternatively, a direct study of the minimum-ehi
square statistic gives additional insight (and amore elementary proof).
As in Chapter 16, say that a sequence of sets Hn converges to a set H if H is the set
of alllimits lim h n of converging sequences h n with h n E Hn for every n and, more over,
the limit h == limi h ni of every converging subsequence h ni with h ni E H ni for every i is
contained in H.
17.4 Theorem. Let Po be a subset of the unit simplex sueh that the sequenee of sets
,Jn (Po - a) converges to a set H (in }Rk), and suppose that a > O. Then, under a,
k ( X . - n p . ) 2 1
. f '"'"' n,l 1 . f X H
ln ln--
PEPO i=l npi hEH -VG
2
for a vector X with the N (O, I - -VG-VG T )-distribution. Here (II -VG)H is the set ofveetors
(hI/,Jiii, . . . , h k /,J7ikJ as h ranges over H.
17.5 Corollary. Let Po be a subset of the unit simplex sueh that the sequenee of sets
,Jn (Po - a) converges to a linear subspaee of dimension 1 (of JRk), and let a > O. Then
both the sequenee of minimum-ehi square statistics and the sequenee of modified Pearson
statisties C n (an) eonverge in distribution to the ehi-square distribution with k -1-1 degrees
of freedom.
Proof. Because the minimum-ehi square estimator a n (relative to P o) is ,Jn-consistent,
the asymptotic distribution of th minimum-ehi square statistic is not changed if we replace
n a n,i in its denominator by the true value nai. Next, we decompose,
,Jn(Pi - ai)
va;
X . -n p '
n,l 1
X . -na.
n,l 1
The first vector on the right converges in distribution to X. The (modified) minimum-ehi
square statistics are the distances of these vectors to the sets Hn == ,Jn (Po - a) / -VG, which
converge to the set H I -VG. The theorem now follows from Lemma 7.13.
The vec tor X is distributed as Z - Il.ja Z for Il.ja the projection onto the linear space
spanned by the vector -VG and Z a k-dimensional standard normal vector. Because every
element of H is the limit of a multiple of differences of probability vectors, 1 T h == O for
every h EH. Therefore, the space (1/ -VG) H is orthogonal to the vector -VG, and Il n.ja == O
for TI the projection onto the space (1 I -VG) H. The distance of X to the space (1/ -VG) H is
equal to the norm of X - IlX, which is distributed as the norm of Z - Il.jaZ - nz. The
latter projection is multivariate normally distributed with mean zero and covariance matrix
the projeetion matrix I - Il.ja - fI with k - 1 - 1 eigenvalues 1. The corollary follows
from Lemma 17.1 or 16.6. .
17.4 Testing Independence
247
17.6 Example (Parametric model). If the nulI hypothesis is a parametric family Po ==
{Pe : f) E e} indexed by a subset e of IRi with I < k and the maps e 1---+ Pe from e
into the unit simplex are differentiable and of fuH rank, then -Jfi (Po - Pe) P e (IRi) for
o
everye E e (see Example 16.11). Then the chi-square statistics C n (Pe) are asymptotically
xl-i-l -distributed.
This situation is common in testing the goodness-of- fit of parametric families, as dis-
cussed in section 17.5 and Example 16.1. D
17.4 Testing Independence
Suppose that each element of a population can be classified according to two characteristics,
having k and r levels, respectively. The full information concerning the classification can
be given by a (k x r) table of the form given in Table 17.1.
Often the full informatian is not available, but we do know the classification Xn,ij for a
random sample of size n from the population. The matrix Xn,ij, which can also be written
in the form of a (k x r) table, is multinomially distributed with parameters n and probabil-
ities Pij == Ni} I N. The nulI hypothesis of independence asserts that the two categories are
independent: Ha: Pij == ai b j for (unknown) probability vectors ai and b j.
The maximum likelihood estimators for the parameters a and b (under the null hypothe-
sis) are ai == Xn,i.1 n and b j == Xn,.j I n. With these estimators the modified Pearson statistic
takes the fonn
kr,," 2
Cn({Zn 0 h n ) = L L (X n . ij -=- aibj) .
i=l j=l naibj
The null hypothesis is a (k + r - 2)-dimensional submanifold of the unit simplex in IR kr .
In a shrinking neighborhood of a parameter in its interior this manifold looks like its
tangent space, a linear space of dimension k + r - 2. Thus, the sequence C n (an Q9 b n ) is
asymptotically chi square-distributed with kr - 1 - (k + r - 2) == (k - l)(r - 1) degrees
of freedom.
Table 17.1. Classijication of a population
of N elements according to two categories,
Ni} elements having value i on the first
category and value j on the second. The
borders give the sums over each row and
column, respectively.
Nu N 12 N Ir NI.
N 2I N 22 N Ir N 2 .
Nk! N k2 N Ir Nk.
N. I N. 2 N. r N
248
Chi-Square Tests
17.7 Corollary. Ifthe (k x r) matrices X n are multinomially distributed with parameters
n and Pij == aib j > 0, then the sequence C n (an 0 bn) converges in distribution to the
X1k-I)(r-1) -distribution.
Proof. The map (al, . . . , ak-I, bI, . . . , br-I) f-* (a x b) from JRk+r-2 into JRkr is con-
tinuously differentiable and of full ranko The true values (al, . . . , ak-I, bI . . . , br-I) are
interior to the domain of this map. Thus the sequence of sets ,Jn (Po - a x b) converges
to a (k + r - 2)-dimensionallinear subspace of JRkr. .
*17.5 Goodness-of-Fit Tests
Chi-square tests are often applied to test goodness-of-fit. Given arandom sample XI, . . . , X n
from a distribution P, we wish to test the nulI hypothesis Ha : P E Po that P belongs to
a given class Po of probability measures. There are many possible test statistics for this
problem, and a particular statistic might be selected to attain high power against certain
alternatives. Testing goodness-of-fit typicalIy focuses on no particular alternative. Then
ehi-square statistics are intuitively reasonable.
The data can be reduced to a multinomial vector by "grouping." We choose a partition
X == U jX j of the sample space into finitely many sets and base the test only on the observed
numbers of observations falling into each of the sets X j . For ease of notation, we express
these numbers into the empirical measure of the data. For a given set A we den ote by
IP n (A) == n-I (1 < i < n : Xi E A) the fraction of observations that fall into A. Then the
vec tor n (IPn (XI), . . . , IP n (X k ) ) possesses a multinomial distribution, and the corresponding
modified ehi -square statistic is given by
k ( A ) 2
2: n JP'n(X ) - P(X j ) .
i=l P(X j )
Here P (X j ) is an estimate of P (X j ) under the null hypothesis and can take a variety of
forms.
Theorem 17.4 applies but is restricted to the case that the estimates P (X j ) are based
on the frequencies n(IP n (X 1 ), . .., IPn(X k )) only. In the present situation it is more natural
to base the estimates on the original observations XI, . . . , X n. U sualIy, this results in a
non-ehi square limit distribution. For instance, Table 17.2 shows the "errors" in the level of
a ehi -square test for testing normality, if the unknown mean and variance are estimated by
the sample mean and the sample variance but the critical value is chosen from the ehi -square
distribution. The size of the errors depends on the numbers of cells, the errors being small
if there are many cells and few estimated parameters.
17.8 Example (Parametric model). Consider testing the nun hypothesis that the true dis-
tribution belongs to a regular parametric model {Pe : e E 8}. It appears natural to estimate
the unknown parameter e by an estimator en that is asymptotically efficient under the nuU
hypothesis and is based on the original sample XI, . . . , X n, for instance the maximum
likelihood estimator. If G n == ,Jn (IP n - Pe) denotes the empirical process, then efficiency
entails the approximation ,Jn(e n - e) == Ie-I GnR-e + op(l). Applying the delta method to
17.5 Goodness-of-Fit Tests
249
Table 17.2. True levels of the chi-square test for normality using
Xf-3,a -quantiles as critical values but estimating unknown mean
and variance by sample mean and sample variance. Chi square
statistic based on partitions of [ -10, 10] into k == 5, 10, ar 20
equiprobable cells under the standard normallaw.
a == 0.20 a==0.10 a == 0.05 a == 0.01
k==5 0.30 0.15 0.08 0.02
k == 10 0.22 0.11 0.06 0.01
k == 20 0.21 0.10 0.05 0.01
Note: Values based on 2000 simulations of standard normal samples of size
100.
the variables -Jii(Pg (X}) - Pe (X}») and using Slutsky's lemma, we find
-Jii(IPn (X}) - Pg (X}»)
J Pe(X})
. T -1 .
CG n Ix. - (Pe 1x.-€ e) l e CGn-€ e
} } + op(l).
! Pe(X})
. .
(The map e 1---+ Pe(A) has derivative Pe 1A-€e.) The sequence of vectors (CG n Ix), CGn-€e)
converges in distribution to a multivariate-normal distribution. Some matrix manipulations
show that the vectors in the preceding display are asymptotically distributed as a Gaussian
vector X with mean zero and covariance matrix
e 1vf e .
(Ce)ij = J(:e) j" .
In general, the covariance matrix of X is not a projection matrix, and the variable II X 11 2
does not possess a ehi-square distribution.
Because Pe-€e == O, we have that Ce,JQe == O and henee the eovariance matrix of X can
be rewritten as the product (1 - ,JQe,JQeT)(/ - Cl le- l Ce). Here the first matrix is the
projection onto the orthocomplement of the veetor ,JQe and the seeond matrix is a positive-
definite transformation that leaves ,JQe invariant, thus aeting only on the orthocomplement
,JQe 1... This geometric picture shows that Cove X has the same system of eigenvectors as the
matrix / - Cl le- l Ce, and also the same eigenvalues, exeept for the eigenvalue eorresponding
to the eigenveetor ,JQe, whiehis O forCove X and 1 for 1 -CllilC e . Because bothmatrices
Cl le- l Ce and / - Cl le- l Ce are nonnegative-definite, the eigenvalues are contained in
[O, 1]. One eigenvalue (eorresponding to eigenveetor ,JQe) is O, dim N(C e ) -1 eigenvalues
(corresponding to eigenspaee N (Ce) n ,JQe..l) are 1, but the other eigenvalues may be
eontained in (O, 1) and then typieally depend on e. By Lemma 17.1, the variable IIXII 2 is
distributed as
1 - ,JQe T - cl le- l Ce,
(ae)} == Pe(X}),
dimN(Ce)-l k-l
L zf + L Ai(e)zf.
i=l i=dimN(C e )
This means that it is stochastieally "between" the ehi-square distributions with dim N (Ce) -
1 and k - 1 degrees of freedom.
The inconvenienee that this distribution is not standard and depends on e can be remedied
by not using effieient estimators en or, altematively, by not using the Pearson statistic.
250
Chi-Square Tests
The square root of the matrix 1 - C[ le- 1 Ce is the positive-definite matrix with the same
eigenvectors, but with the square roots of the eigenvalues. Thus, it also leaves the vector
-JZi8 invariant and acts only on the orthocomplement -JZi81... It follows that this square root
commutes with the matrix 1 - -JZi8 -JZi8T and hence
( I - cT 1:: 1 c ) -1/2 vfn(JP'n (X j ) - PiJ (X j )) 'v-7 N ( O 1 _ ra: ra:T ) .
e e e YI Pe (X j ) k, V ue V ue
(We assume that the matrix 1 - C[ le- 1 Ce is nonsingular, which is typically the case; see
problem 17.6). By the continuous-mapping theorem, the squared norm of the left side is
asymptotically chi square-distributed with k - 1 degrees of freedom. This squared norm
is the Rao-Robson-Nikulin statistic. D
It is tempting to choose the partitioning sets X j dependent on the observed data XI, . . . ,
X n , for instance to ensure that all cells have positive probability under the nulI hypothesis.
This is permissible under some conditions: The choice of a "random partition" typically
does not change the distributional properties of the ehi-square statistic. Consider partition-
ing sets X j == X j (X 1, . . . , X n ) that possibly depend on the data, and a further modified
Pearson statistic of the type
t n(JP'n (X ) --= p(X j ) )2 .
i=l P(X j )
If the random partitions settle down to a fixed partition eventually, then this statistic is
asymptoticaUy equivalent to the statistic for which the partition had been set equal to the
limit partition in advance. We discuss this for the case that the nuU hypothesis is amodel
{Pe : e E e} indexed by a subset e of a normed space. We use the language of Donsker
classes as discussed in Chapter 19.
17.9 Theorem. Suppose that the sets X j belong to a Peo-Donsker class C of sets and
" p
that Pe o (X j 6. X j ) O under Pe o , for given nonrandom sets X j such that Peo (X j ) > o.
Furthermore, assume that vfnlle - eoll == Op(l), and suppose that the map e r-+ Pe from
e into fOO(C) is differentiable at e o with derivative Pe o such that Peo(X j ) - Peo(X j ) O
for every j. Then
k ( " ,, ) 2 k ( ) 2
"n Pn(X j ) - Pe(X j ) "n Pn(X j ) - Pe(X j )
" == + op(l).
i=l Pe (X j ) i=l Pe (X j )
Proof. Let (Q.n == vfn(Pn - Peo) be the empirical process and define IHIn == vfn(Pe - P eo ).
Then vfn(pn(Xj) - Pe(X j )) == ((Q.n - IHIn)(X j ), and similarly with X j replacing Xj.
The condition that the sets X j belong to a Donsker class cornbined with the continuity
condition PeoCXj 6. Xj) O, imply that (Q.n(X j ) - CGn(X j ) O (see Lernma 19.24). The
differentiability of the map e r-+ Pe implies that
sup Ipe(C) - Peo(C) - Peo(C)(e - eo)1 == op(lle - (1 0 11).
c
Together with the continuity P e o (X j ) - P eo (X j ) O and the vfn-consistency of e, this
17.6 Asymptotic Efficiency
251
A P A P A
shows that JH[n (X}) - IHIn (X}) --+ O. In particular, because Peo (X}) --+ Pe o (X}), both Pe (X})
and Pe(X}) converge in probability to Peo(X}) > O. The theorem follows. .
The conditions on the random partitions that are imposed in the preceding theorem
are mild. An interesing choice is a partition in sets X} (e) such that Pe (X) (e)) == a} is
independent of e. The corresponding modified Pearson statistic is known as the Watson-Roy
statistic and takes the form
k n ( JP' n ( X j (e)) - aj f
L .
i=l a}
Here the null probabilities have been reduced to fixed values again, but the cell frequencies
are "doubly random." If the model is smooth and the parameter and the sets X} (e) are
not too wild, then this statistic has the same nulllimit distribution as the modified Pearson
statistic with a fixed partition.
17.10 Example (Location-scale). Consider testing a null hypothesis that the true under-
lying measure of the observations belongs to a location-scale family { Fo ( (. - J-l) / a) : J-l E
IR, a > O}, given a fixed distribution Fo on JR. It is reasonable to choose a partition in sets
X} == {L + a (c }-l, c)], for a fixed partition -00 == Co < Cl < . . . < Ck == 00 and estimators
IL and a of the location and scale parameter. The partition could, for instance, be chosen
equal to c} == FO-l (j / k), although, in general, the partition should depend on the type of
deviation from the null hypothesis that one wants to detect.
If we use the same location and scale estimators to "estimate" the null probabilities
Fo ( (X) - J-l) / a) of the random cells X} == Il + a (c) -1, c}], then the estimators cancel, and
we find the fixed null probabilities Fo (c}) - Fo (c) -1). D
* 17.6 Asymptotic Efficiency
The asymptotic null distributions of various versions of the Pearson statistic enable us to
set critical values but by themselves do not give information on the asymptotic power of
the tests. Are these tests, which appear to be mostly motivated by their asymptotic null
distribution, sufficiently powerful?
The asymptotic power can be measured in various ways. Probably the most important
method is to consider locallimiting power functions, as in Chapter 14. For the likelihood
ratio test these are obtained in Chapter 16. Because, in the local experiments, ehi-square
statistics are asymptotically equivalent to the likelihood ratio statistics (see Theorem 17.4),
the results obtained there also apply to the present problem, and we shall not repeat the
discussion.
A second method to evaluate the asymptotic power is by Bahadur effieiencies. For this
nonlocal criterion, ehi-square tests and likelihood ratio tests are not equivalent, the second
being better and, in fact, optimal (see Theorem 16.12).
We shall eompute the slopes of the Pearson and likelihood ratio tests for testing the simple
hypothesis Ho : p == a. A multinomial vec tor X n with parameters n and p == (Pl, . . . , Pk)
ean be thought of as n times the empirical measure JP> n of arandom sample of size n from
the distribution P on the set {I, . . . , k} defined by P {i} == Pi. Thus we can view both the
252
Chi-Square Tests
Pearson and the likelihood ratio statistics as functions of an empirical measure and next
can apply Sanov's theorem to compute the desired limit s of large deviations probabilities.
Define maps C and K by
C(p,a) = t (Pi -a J2 ,
i=1 ai
k
a Pi
K(p,a) == -Plog- == LPilog-.
P i=1 ai
Then the Pearson and likelihood ratio statistics are equivalent to C (JED n, a) and K (P n, a),
respectively.
Under the assumption that a > O, both maps are continuous in P on the k-dimensional
unit simplex. Furthermore, for t in the interior of the ranges of C and K, the sets Bt ==
{p: C(p, a) > t} and Et == {p: K(p, a) > t} are equal to the closures of the ir interiors.
Two applications of Sanov's theorem yield
1
- log Pa ( C (JED n, a) > t) --+ - inf K (p, a),
n pE
1
-logPa(K(JED n , a) > t) --+ - inf K(p, a) == -to
n pE
We take the function e(t) of (14.20) equal to minus two times the right sides. Because
Pn{i} --+ Pi by the law of large numbers, whence C(P n , a) C(P, a) and K(Pn, a)
K (P, a), the Bahadur slopes of the Pearson and likelihood ratio tests at the alternative
HI : P == q are given by
2 inf K (p, a)
p:C(p,a) C(q ,a)
and
2K(q, a).
It is clear from these expressions that the likelihood ratio test has a bigger slope. This is
in agreement with the fact that the likelihood ratio test is asymptoticalIy Bahadur optimal
in any smooth parametric model. Figure 17.1 shows the difference of the slopes in one
particular case. The difference is smalI in a neighborhood of the nulI hypothesis a, in
agreement with the fact that the Pitman efficiency is equal to 1, but can be substantial for
a1ternatives away from a.
Notes
Pearson introduced his statistic in 1900 in [112] The modification with estimated para-
meters, using the mu1tinomial frequencies, was considered by Fisher [49], who corrected
the mistaken belief that estimating the parameters does not change the limit distribution.
Chernoff and Lehmann [22] showed that using maximum likelihood estimators based on
the original data for the parameter in a goodness-of-fit statistic destroys the asymptotic
ehi -square distribution. They note that the errors in the level are smalI in the case of testing
a Poisson distribution and somewhat larger when testing normality.
Problems
253
Lf)
C\I
C\I
o
Qa
.,....
.,....
Lf)
o
Qe
O.CO
Q<1
Q<2
O.tIt
o.
Figure 17.1. The difference of the Bahadur slopes of the likelihood ratio and Pearson tests for testing
Ha : p == (1/3, 1/3, 1/3) based on a multinomial vector with parameters n and p == (Pl, P2, P3), as
a function of (p I, P2).
The ehoiee of the partition in ehi-square goodness-of-fit tests is an important issue that we
have not diseussed. Several authors have studied the optimal number of eells in the partition.
This number depends, of eourse, on the alternative for whieh one desires large power. The
eonelusions of these studies are not easily summarized. For alternative s p sueh that the
likelihood ratio p / peo with respeet to the nulI distribution is "wild," the number of eells k
should tend to infinity with n. Then the ehi-square approximation of the nulI distribution
needs to be modified. Normal approximations are used, beeause a ehi-square distribution
with a large number of degrees of freedom is approximately a normal distribution. See [40],
[60], and [86] for results and further referenees.
PROBLEMS
1. Let N == (Ni}) be a multinomial matrix with success probabilities Pij. Design a test statistic for
the null hypothesis of symmetry Ha : Pij == P ji and derive its asymptotie null distribution.
2. Derive the limit distribution of the chi-square goodness-of-fit statistic for testing normality if
using the sample mean and sample variance as estimators for the unknown mean and variance.
U se two or three cells to keep the calculations simple. Show that the limit distribution is not
chi-square.
3. Suppose that X m and Y n are independent multinomial vectors with parameters (m, al, . . . , ak)
and (n, bI, . . . , bk), respectively. Under the null hypothesis Ha : a == b, a natural estimator of
the unknown probability vec tor a == b is C == (m + n)-I (X m + Y n ), and a natural test statistic is
given by
(Xmi - mc i)2 (Yni - nc i)2
' + '
. I mCi . I nCi
l= l=
Show that C is the maximum likelihood estimator and show that the sequence of test statistics is
asymptotically ehi square-distributed if m, n --+ 00.
254
Chi-Square Tests
4. A matrix L: - is called a generalized inverse of a matrix L: if x == L: - Y solves the equation
L:x == y for every y in the range of L:. Suppose that X is Nk (O, L:)-distributed for a matrix L: of
rank r. Show that
(i) yT L:- Y is the same for every generalized inverse L:-, with probability one;
(ii) yT L: - Y possesses a ehi-square distribution with r degrees of freedom;
(iii) if Y T C y possesses a chi -square distribution with r degrees of freedom and C is a nonnegative-
definite symmetric matrix, then C is a generalized inverse of L:.
5. Pind the limit distribution of the Dzhaparidze-Nikulin statistic
(1P n (X j ) - Po (X j )) ( T ( T ) -I ) (Pn(Xj) - Po (X}))
n I - CA CACA CA
J Po (X j ) e e e e J Po (X}) .
6. Show that the matrix I - Cl Ie- I Ce in Example 17.8 is nonsingular unless the empirical estima-
tor (JID n (XI), . . . , JID n (X k )) is asymptotically efficient. (The estimator (po (XI), . . . , Po (X k )) is
asymptotically efficient and has asymptotic covariance matrix diag (,jae) Cl Ie- I Ce diag (,Jfie);
the empirical estimator has asymptotic covariance matrix diag (,Jfie) (I - ,Jfie ,JfieT) diag (,Jfie).)
18
Stochastic Convergence in Metric Spaces
This chapter extends the concepts of convergence in distribution, in prob-
abi/ity, and almost surely from Euclidean spaces to more abstract metric
spaces. We are particularly interested in developing the theory for ran-
dom functions, or stochastic processes, viewed as elements of the metric
space of all bounded functions.
18.1 Metric and Normed Spaces
In this section we recall some basic topological concepts and introduce a number of examples
of metric spaces.
A metric space is a set JI]) equipped with a metric. A metric or distance function is a map
d : [J) x [J) 1-+ [O, (0) with the properties
(i) d(x, y) == d(y, x);
(ii) d (x, z) < d (x, y) + d (y, z) (triangle inequality);
(iii) d (x, y) == O if and only if x == y.
A semimetric satisfies (i) and (ii), but not necessarily (iii). An open ball is a set of the
form {y : d (x, y) < r}. A subset of a metric space is open if and only if it is the union of
open balls; it is closed if and only if its complement is open. A sequence x n converges to x if
and only if d (x n , x) ---+ O; this is denoted by x n ---+ x. The closure A of a set A C [J) consists
of all points that are the limit of a sequence in A; it is the smallest closed set containing A.
The interior Ji is the collection of all points x such that x E GeA for some open set G;
it is the largest open set contained in A. A function f : [J) 1-+ JE between two metric spaces
is continuous at a point x if and only if f (x n ) ---+ f (x) for every sequence X n ---+ x; it is
continuous at every x if and only if the inverse image f-l (G) of every open set G c JE
is open in [J). A subset of a metric space is dense if and only if its closure is the whole
space. A metric space is separable if and only if it has a countable dense subset. A subset
K of a metric space is compact if and only if it is closed and every sequence in K has a
converging subsequence. A subset K is totally bounded if and only if for every 8 > O it
can be covered by finitely many balls of radius 8. A semimetric space is complete if every
Cauchy sequence, a sequence such that d(xn, X m ) ---+ O as n, m ---+ 00, has alimit. A subset
of a complete semimetric space is compact if and only if it is totally bounded and closed.
A normed space JI]) is a vector space equipped with a norm. A norm is a map II . II : [J) 1-+
[O, (0) such that, for every x, y in [J), and ex E IR,
255
256
Stochastic Convergence in Metric Spaces
(i) Ilx + y II < Ilx II + II y II (triangle inequality);
(ii) Ilax II == la Illx II;
(iii) Ilx II == O if and only if x == O.
A seminorm satisfies (i) and (ii), but not necessarily (iii). Given a nOfIll, a metric can be
defined by d(x, y) == Ilx - y II.
18.1 Definition. The Borel a -field on a metric space JI)) is the smallest a - field that contains
the open sets (and then also the closed sets). A function defined relative to (one or two)
metric spaces is called Borel-measurable if it is measurable relative to the Barel a-field(s).
A Borel-measurable map X : Q r-+ JI)) defined on a probability space ([2, U, P) is referred
to as arandom element with values in JI)).
For Euclidean spaces, Borel measurability is just the usual measurability. Barel measur-
ability is probably the natural concept to use with metric spaces. It combines well with the
topological structure, particularly if the metric space is separable. For instance, continuous
maps are Borel-measurable.
18.2 Lemma. A continuous map between metric spaces is Borel-measurable.
Proof. A map g : JI)) r-+ lE: is continuous if and only if the inverse image g -1 (G) of every
open set GelE: is open in JI)). In particular, for every open G the set g-l (G) is a Barel set in
JI)). By definition, the open sets in lE: generate the Borel a-field. Thus, the inverse image of a
generator of the Borel sets in lE: is contained in the Borel a-field in JI)). Because the inverse
image g-l (9) of agenerator 9 of aa-field B generates the a-field g-l (8), it follows that
the inverse image of every Borel set is a Borel set. .
18.3 Example (Euclidean spaces). The Euclidean space k is a normed space with re-
spect to the Euclidean norm (whose square is Ilx 11 2 == L =lxl), but also with respect to
many other norms, for instance Ilx II == maxi IXi I, all of which are equivalent. By the Heine-
Borel theorem a subset of k is compact if and only if it is closed and bounded. A Euclidean
space is separable, with, for instance, the vectors with rational coordinates as a countable
dense subset.
The Borel a - field is the usual a - field, generated by the interval s of the type ( - 00, x]. O
18.4 Example (Extended real line). The extended real line == [-00,00] is the set
consisting of all real numbers and the additional elements -00 and 00. It is a metric space
with respect to
d(x, y) == Icl>(x) - <p(y)l.
Here cl> can be any fixed, bounded, strictly increasing continuous function. For instance,
the normal distribution function (with cl> ( - (0) == O and cl> (00) == 1). Convergence of a
sequence x n -* x with respect to this metric has the usual meaning, also if the limit x
is -00 or 00 (normally we would say that x n "diverges"). Consequently, every sequence
has a converging subsequence and hence the extended real line is compact. D
18.1 Metric and Normed Spaces
257
18.5 Example (Uniform norm). Given an arbitrary set T, let fOO(T) be the collection
of all bounded functions Z : T 1---+ JR. Define sums ZI + Z2 and products with scalars
az pointwise. For instance, ZI + Z2 is the element of fOO(T) such that (ZI + Z2)(t)
ZI (t) + Z2(t) for every t. The uniform norm is defined as
IlzIIT == suplz(t)l.
tET
With this notation the space goo (T) consists exactly of all functions z : T 1---+ JR such that
IlzIIT < 00. The space fOO(T) is separable if and only if T is countable. D
18.6 Example (Skorohod space). Let T == [a, b] be an interval in the extended real line.
We den o te by C [a, b] the set of all continuous functions z : [a, b] 1---+ JR and by D [a, b] the set
of all functions Z : [a, b] 1---+ JR that are right continuous and whose limit s from the left exist
everywhere in [a, b]. (The functions in D[a, b] are called cadlag: continue il droite, limites
il gauche.) lt can be shown that C[a, b] c D[a, b] c fOO[a, b]. We always equip the spaces
C[a, b] and D[a, b] with the uniform norm IlzIIT, which they "inherit" from fOO[a, b].
The space D [a, b] is referred to here as the Skorohod space, although Skorohod did not
consider the uniform norm but equipped the space with the "Skorohod metric" (which we
do not use or discuss).
The space era, b] is separable, but the space D[a, b] is not (relative to the uniform
norm). O
18.7 Example (Unifornlly continuous functions). Let T be a totally bounded semimetric
space with semimetric p. We denote by ue (T, p) the collection of all uniformly continuous
functions z : T 1---+ JR. Because a uniformly continuous function on a totally bounded set
is necessarily bounded, the space ue (T, p) is a subspace of foo (T). We equip ue (T, p)
with the uniform norm.
Because a compact sernimetric space is totally bounded, and a continuous function on
a compact space is automatically uniforrn1y continuous, the spaces e (T, p) for a compact
semimetric space T, for instance C[a, b], are special cases of the spaces UC(T, p). Actu-
ally, every space U C (T, p) can be identified with a space e (T, p), because the completion
T of a totally bounded semirnetric T space is compact, and every uniformly continuous
function on T has a unique continuous extension to the completion.
The space UC(T, p) is separable. Furthermore, the Borel a-field is equal to the a-field
generated by all coordinate projections (see Problem 18.3). The coordinate projections
are the maps z 1---+ zet) with t ranging over T. These are continuous and hence always
Borel-measurable. O
18.8 Example (Product spaces). Given a pair of metric spaces JI}) and JE with metrics d
and e, the Cartesian product JI}) x JE is a metric space with respect to the metric
f((XI, YI), (X2, Y2)) == d(XI, X2) V e(YI, Y2).
For this metric, convergence of a sequence (x n , Yn) (x, y) is equivalent to both X n X
and Yn Y.
For a product metric space, there exist two natural a-fields: The product of the Borel
a-fields and the Borel a-field of the product metric. In general, these are not the same,
258
Stochastic Convergence in Metric Spaces
the second one being bigger. A sufficient condition for them to be equal is that the metric
spaces II]) and lE are separable (e.g., Chapter 1.4 in [146])).
The possible inequality of the two o- - fields causes an inconvenient problem. If X : sl JI])
and Y : sl lE are Borel-measurable maps, defined on some measurable space (sl, U), then
(X, Y) : sl JI]) x JE is always measurable for the product of the Borel o--fields. This is
an easy fact from measure theory. However, if the two o- - fields are different, then the map
(X, Y) need not be Borel-measurable. If they have separable range, then they are. D
18.2 Basic Properties
In Chapter 2 convergence in distribution of random vectors is defined by reference to their
distribution functions. Distribution functions do not extend in a natural way to random
elements with values in metric spaces. Instead, we define convergence in distribution using
one of the characterizations given by the portmanteau lemma.
A sequence of random elements X n with values in a metric space II]) is said to converge
in distribution to arandom element X ifEf(X n ) ---+ Ef(X) for every bounded, continuous
function f : JI]) IR:. In some applications the "random elements" of interest tum out not
to be Borel-measurable. To accomodate this situation, we extend the preceding definition
to a sequence of arbitrary maps X n : Sln II]), defined on probability spaces (Sln, Un, Pn).
Because Ef (X n ) need no long er make sense, we replace expectations by outer expectations.
For an arbitrary map X : sl II]), define
E* J(X) == inf {EU: U : sl r-+ IR:, measurable, U > J(X), EU exists}.
Then we say that a sequence of arbitrary maps X n : Sln 1--+ JI]) converges in distribution
to arandom element X if E* J(Xn) --+ Ef(X) for every bounded, continuous function
f : JI]) 1--+ IR:. Here we insist that the limit X be Borel- measurable.
In the following, we do not stress the measurability issues. However, throughout we do
write stars, if necessary, as areminder that there are measurability issues that need to be
taken care of. Although Sln may depend on n, we do not let this show up in the notation
for E* and P* .
Next consider convergence in probability and almost surely. An arbitrary sequence of
maps X n : Sln 1--+ JI]) converges in probability to X if P* (d (X n , X) > £) --+ O for all £ > O.
This is denoted by X n X. The sequence X n converges almost surely to X ifthere exists
a sequence of (measurable) random variables n such that d(Xn, X) < n and n O.
This is denoted by X n X.
These definitions also do not require the X n to be Borel-measurable. In the definition of
convergence of probability we solved this by adding a star, for outer probability. On the
other hand, the definition of almost -sure convergence is unpleasant1y complicated. This
cannot be avoided easily, because, even for Borel-measurable maps X n and X, the distance
d(X n , X) need not be arandom variable.
The portmanteau lemma, the continuous-mapping theorem and the relations among the
three modes of stochastic convergence extend without essential changes to the present defini-
tions. Even the proofs, as given in Chapter 2, do not need essential modifications. However,
we seize the opportunity to formulate and prove a refinement of the continuous- mapping
theorem. The continuous-mapping theorem furnishes amore intuitive interpretation of
18.2 Basic Properties
259
weak convergence in terms of weak convergence of random vectors: X n -v--t X in the metric
space llJ) if and only if g(Xn) g(X) for every continuous map g : [J) 1--* JRk.
18.9 Lemma (Portmanteau). For arbitrary maps X n : S1 n 1--* [J) and every random ele-
ment X with values in llJ), the following statements are equivalent.
(i) E* f(Xn) --+ Ef(X) for all bounded, continuous functions f.
(ii) E* f(Xn) --+ Ef(X) for all bounded, Lipschitzjunctions f.
(iii) liminfP*(X n E G) > P(X E G)foreveryopensetG.
(iv) lim sup P*(X n E F) < P(X E F) for every closed set F.
(v) P*(X n E B) --+ P(X E B)forall Borel sets B with P(X E 8B) == O.
18.10 Theorem. For arbitrary maps X n , Y n : n 1--* llJ) and every random element X with
values in [J):
(l ' ) X as* X . 1 . X p X
n --+ lmp les n --+ .
(ii) X n X implies X n -v--t X.
(iii) X n c for a constant c if and only if X n -v--t c.
(iv) if X n -v--t X and d(Xn, Y n ) O, then Y n X.
(v) if X n -v--t X and Y n cfor a constant c, then (X n , Y n ) -v--t (X, c).
(vi) if X n X and Y n Y, then (X n , Y n ) (X, Y).
18.11 Theorem (Continuous mapping). Let llJ)n C [J) be arbitrary subsets and gn : llJ)n 1--*
JE be arbitrary maps (n > O) such that for every sequence X n E [J)n : if X n ' --+ X along a
subsequence and X E llJ)o, then gn' (xn') --+ go(x). Then, for arbitrary maps X n : S1 n 1--* llJ)n
and every random element X with values in [J)o such that go (X) is arandom element in JE:
(i) If X n -v--t X, then gn (X n ) -v--t go(X).
(ii) If X n X, then gn(X n ) go(X).
(iii) If X n X, then gn (X n ) go(X).
Proof. The proofs for II}n == II} and gn == g fixed, where g is continuous at every point of
II}o, are the same as in the case of Euclidean spaces. We prove the refinement only for (i).
The other refinements are not needed in the following.
For every closed set F, we have the inclusion
n lU =k{X E II}m: gm(x) E F} C gol(F) U ([J) - [J)o).
lndeed, suppose that X is in the set on the left side. Then for every k there is an mk > k and
an element x mk E g;;;;(F) with d(x mk , x) < lik. Thus, there exist a sequence mk --+ 00
and elements x mk E [J)mk with x mk --+ x. Then either gmk (x mk ) --+ go (x) or X [J)o. Because
the set F is closed, this implies that go (x) E F or X [J)o.
Now, for every fixed k, by the portmanteau lemma,
limsupP*(gn(X n ) E F) < limsupP* ( X n E U =k{X E [J)m :gm(x) E F} )
n ----+ 00 n ----+ 00
< p(X E U =kg;l(F)).
260
Stochastic Convergence in Metric Spaces
As k --+ 00, the last probability converges to p(X E n l U =k g;;;l (F)), which is smaller
than or equal to p(go(X) EF), by the preceding paragraph. Thus, gn (X n ) 'v'7 go(X) by the
portmanteau lemma in the other direction. .
The extension of Prohorov's theorem requires more care. t In a Euclidean space, a set
is compact if and only if it is closed and bounded. In general metric spaces, a compact
set is closed and bounded, but a closed, bounded set is not necessarily compact. It is the
compactness that we employ in the definition of tightness. A Borel-measurable random
element X into a metric space is tight if for every £ > O there exists a compact set K such
that P(X ti- K) < £. A sequence of arbitrary maps X n : n 1---+ JI)) is called asymptotically
tight if for every £ > O there exists a compact set K such that
lim sup P* (X n ti- K 8 ) < £,
n ---+ 00
every 8 > O.
Here K 8 is the 8-enlargement {y: d(y, K) < 8} of the set K. It can be shown that, for
Borel-measurable maps in IR k , this is identical to "uniformly tight," as defined in Chapter 2.
In order to obtain a theory that applies to a sufficient number of applications, again we do
not wish to assume that the X n are Borel-measurable. However, Prohorov's theorem is true
only under, at least, "measurability in the limit." An arbitrary sequence of map s X n is called
asymptotically measurable if
E* f(Xn) - E*f(X n ) --+ O,
every f E Cb(D).
Here E* denotes the inner expectation, which is defined in analogy with the outer expec-
tation, and C b (JI))) is the collection of all bounded, continuous functions f: JI)) 1---+ IR. A
Borel-measurable sequence of random elements X n is certainly asymptotically measur-
able, because then both the outer and the inner expectations in the preceding display are
equal to the expectation, and the difference is identically zero.
18.12
Theorem (Prohorov's theorem). Let X n : n --+ JI)) be arbitrary maps into a metric
space.
(i)
( ii)
If X n 'v'7 X for some tight random element X, then {X n : n E N} is asymptotically
tight and asymptotically measurable.
If X n is asymptotically tight and asymptotically measurable, then there is a subse-
quence and a tight random element X such that X nj 'v'7 X as i --+ 00.
18.3 Bounded Stochastic Processes
A stochastic process X == {X t : t ET} is a collection of random variables X t : Q 1---+ IR,
indexed by an arbitrary set T and defined on the same probability space ( , U, P). For
a fixed w, the map t 1---+ X t (w) is called a sample path, and it is helpful to think of X as
arandom function, whose realizations are the sample paths, rather than as a collection of
random variables. If every sample path is a bounded function, then X can be viewed as a
t The following Prohorov's theorem is not used in this book. For a proof see, for instance, [146].
18.3 Bounded Stochastic Processes
261
map X : Q loo (T). If T == [a, b] and the sample paths are continuous or cadlag, then X
is also a map with values in C[a, b] or D[a, b].
Because C[a, b] c D[a, b] c lOO[a, b], we can consider the weak convergence of a
sequence of maps with values in C[a, b] relative to C[a, b], but also relative to D[a, b], or
loo [a, b]. The following lemma shows that this does not make a difference, as long as we
use the uniform norm for all three spaces.
18.13 Lemma. Let [J)o C II]) be arbitrary metric spaces equipped with the same metric. If
X and every X n take their values in ITJ)o, then X n X as maps in ITJ)o if and only if X n X
as maps in ITJ).
Proof. Because a set Go in ITJ)o is open if and only if it is of the form G n ITJ)o for an open
set G in ITJ), this is an easy corollary of (iii) of the portmanteau lemma. .
Thus, we may concentrate on weak convergence in the space lOO(T), and automatically
obtain characterizations of weak convergence in C[a, b] or D[a, b]. The next theorem
gives a characterization by finite approximation. It is required that, for any £ > O, the index
set T can be partitioned into finitely many sets TI, . . . , Tk such that (asymptotically) the
variation of the sample paths t Xn,t is less than £ on every one of the sets yi, with large
probability. Then the behavior of the process can be described, within a small error margin,
by the behavior of the marginal vectors (X n ,tl' . . . , Xn,tk) for arbitrary fixed points ti E yi.
If these marginals converge, then the processes converge.
18.14 Theorem. A sequence of arbitrary maps X n : Qn lOO(T) converges weakly to a
tight random element if and only if both of the following conditions hold:
(i) The sequence (X n ,tl' . . . , Xn,tk) converges in distribution in JRk for every finite set of
points tI, . . . , tk in T;
(ii) for every £, rJ > O there exists a partition of T into finitely many sets TI, . . . , Tk
such that
lim sup P* ( s p sup IXn,s - Xn,t I > £ ) < rJ
n-+oo l s,tE1i
Proof. We only give the proof of the more constructive part, the sufficiency of (i) and
(ii). For each natural number m, partition T into sets Tlm, . . . , Tk ' as in (ii) corresponding
to £ == rJ == 2- m . Because the probabilities in (ii) decrease if the partition is refined, we
can assume without loss of generality that the partitions are successive refinements as m
increases. For fixed m define a semimetric Pm on T by Pm (s, t) == O if s and t belong
to the same partioning set T j m , and by Pm (s, t) == 1 otherwise. Every Pm -ball of radius
O < £ < 1 coincides with a partitioning set. In particular, T is totally bounded for Pm,
and the Pm -diameter of a set T;1 is zero. By the nesting of the partitions, Pl < P2 < . . '.
Define p(s, t) == L =12-mpm(s, t). Then P is a semimetric such that the p-diameter of
T;n is smaller than Lk>m 2- k == 2- m , and hence T is totally bounded for p. Let To be the
countable p-dense subset constructed by choosing an arbitrary point tj from every T;n.
By assumption (i) and Kolmogorov's consistency theorem (e.g., [133, p. 244] or [42,
p. 347]), we can construct a stochastic process {X t : t E To} on some probability space such
that (Xn,tl ' . . . , Xn,tk) (X tl , . . . , X tk ) for every finite set of points tI, . . . , tk in To. By the
262
Stochastic Convergence in Metric Spaces
portmanteau lemma and assumption (ii), for every finite set S eTo,
p sup sup IX s - Xtl > 2- m < 2- m .
j s,tETF
s,tES
B y the monotone convergence theorem this remains true if S is replaced by To. If p (s, t) <
2- m , then Pm (s, t) < 1 and hence s and t belong to the same partitioning set T j m . Conse-
quently, the event in the preceding display with S == To contains the event in the following
display, and
p sup IX s - Xtl > 2- m < 2- m .
p(s,t)<2- m
s, t E To
This sums to a finite number over m E N. Rence, by the Borel-Cantelli lemma, for almost all
w, I Xs (w) - Xt (w) I < 2- m for all p (s, t) < 2- m and all sufficient1y large m. This implies
that almost all sample paths of {X t : t E To} are contained in U C (To, p). Extend the process
by continuity to a process {X t : t ET} with almost all sample paths in ue (T, p).
Define Jr m : T f-+ T as the map that maps every partioning set T j m onto the point tj E T j m .
Then, by the uniform continuity of X, and the fact that the p-diameter of T j m is smaller
than 2- m , X o Jr m X in £OO(T) as m -+ 00 (even almost surely). The processes {X n o
Jr m (t) : t ET} are essentially km-dimensional vectors. By (i), X n OJr m X OJr m in £OO(T) as
n -+ 00, for every fixed m. Consequently, for every Lipschitz function f : £00 (T) 1---+ [O, 1],
E* f (X n o Jr m ) -+ Ef (X) as n -+ 00, followed by m -+ 00. Conclude that, for every 8 > O,
IE*f(X n ) -Ef(X)1 < IE*f(X n ) -E*f(X n O Jrm)1 +0(1)
< IIflhi p 8 + P*(IIX n - X n o JrmllT > 8) + 0(1).
For 8 == 2- m this is bounded by II f 11 1ip 2- m + 2- m + 0(1), by the construction of the
partitions. The proof is complete. .
In the course of the proof of the preceding theorem a semimetric p is constructed such
that the weak limit X has uniformly p-continuous sample paths, and such that (T, p) is
totally bounded. This is surprising: even though we are discussing stochastic processes with
values in the very large space £OO(T), the limit is concentrated on a much smaller space of
continuous functions. Actually, this is a consequence of imposing the condition (ii), which
can be shown to be equivalent to asymptotic tightness. It can be shown, more generally, that
every tight random element X in £00 (T) necessarily concentrates on ue (T, p) for some
semimetric p (depending on X) that makes T totally bounded.
In view of this connection between the partitioning condition (ii), continuity, and tight-
ness, we shalI sometimes refer to this condition as the condition of asymptotic tightness or
asymptotic equicontinuity.
We record the existence of the semimetric for later reference and note that, for a Gaussian
limit process, this can always be taken equal to the "intrinsic" standard deviation semimetric.
18.15 Lemma. Under the conditions (i) and (ii) of the preceding theorem there exists
a semimetric p on T for which T is totally bounded, and such that the weak limit of the
Problems
263
sequence X n can be constructed to have almost all sample paths in ue (T, p). Furthermore,
if the weak limit X is zero-mean Gaussian, then this semimetric can be taken equal to
p(s, t) == sd(X s - X t ).
ProoJ. A semimetric p is constructed explicitly in the proof of the preceding theorem. It
suffices to prove the statement conceming Gaussian limits X.
Let p be the semimetric obtained in the proof of the theorem and let P2 be the stan-
dard deviation semimetric. Because every uniformly p-continuous function has a unique
continuous extension to the p-completion of T, which is compact, it is no loss of gener-
ality to assume that T is p-compact. Furthermore, assume that every sample path of X is
p -continuous.
An arbitrary sequence t n in T has a p-converging subsequence t n , ---+ t. By the p-
continuity of the sample paths, X tnl ---+ Xt almost surely. Because every Xt is Gaussian,
this implies convergence ofmeans and variances, whence P2(t n " t)2 == E(X tnl - Xt)2 ---+ O
by Proposition 2.29. Thus t n , ---+ t also for P2 and hence T is P2 -compact.
Suppose that a sample path t Xt (w) is not P2-continuous. Then there exists an
E > O and atE T such that P2(t n , t) -+ O, but IX tn (w) - Xt(w)1 > E for every n. By
the p-compactness and continuity, there exists a subsequence such that p (t n " s) -+ O and
X tn, (w) -+ X s (w) for some s. By the argument of the preceding paragraph, P2 (t n " s) -+ O,
so that P2(S, t) == O and IXs(w) - Xt(w)\ > E. Conclude that the path t Xt(w)
can only fail to be P2 -continuous for w for which there exist s, t E T with P2 (s , t) == O,
but Xs(w) i= Xt(w). Let N be the set of w for which there do exist such s, t. Take a
countable, p-dense subset A of {(s, t) E T x T: P2(S, t) == O}. Because t Xt(w) is P-
continuous, N is also the set of all w such that there exist (s, t) E A with X s (w) i= X t (w).
From the definition of P2, it is clear that for every fixed (s, t), the set of w such that
X s (w) i= X t (w) is a nun set. Conclude that N is a nulI set. Rence, almost all paths of X are
P2 -continuous. .
Notes
The theory in this chapter was developed in increasing generality over the course of many
years. Work by Donsker around 1950 on the approximation of the empirical process and
the partial sum process by the Brownian bridge and Brownian motion processes was an
important motivation. The first type of approximation is discussed in Chapter 19. For
further details and references conceming the material in this chapter, see, for example, [76]
or [146].
PROBLEMS
1. (i) Show that a compact set is totally bounded.
(ii) Show that a compact set is separable.
2. Show that a function f : II]) r+ E is continuous at every X E II]) if and only if f-l (G) is open in II])
for every open G E E.
3. (Projection a-field.) Show that the a-field generated by the coordinate projections Z r+ zet) on
C[a, b] is equal to the Borel a-field generated by the uniform norm. (First, show that the space
264
Stochastic Convergence in Metric Spaces
C[a, b] is separable. Next show that every open set in a separable metric space is a countable
union of open balls. Next, it suffices to prove that every open ball is measurable for the projection
(J -field.)
4. Show that D[a, b] is not separable for the uniform norm.
5. Show that every function in D [a, b] is bounded.
6. Let h be an arbitrary element of D[ -00,00] and let £ > O. Show that there exists a grid ua ==
-00 < Ul < . . . Um == 00 such that h varies at most £ on every interval [Ui, Ui+l). Here "varies
at most £" means that I h (u) - h (v) I is less than £ for every u, v in the interval. (Make sure that
all points at which h jumps more than £ are grid points.)
7. Suppose that Hn and Ha are subsets of a semimetric space H such that Hn ---+ Ha in the sense that
(i) Every h E Ha is the limit of a sequence h n E Hn;
(ii) If a subsequence h nj converges to alimit h, then h E Ha.
Suppose that An are stochastic processes indexed by H that converge in distribution in the
space lOO(H) to a stochastic process A that has uniformly continuous sample paths. Show that
SUPhEH n An (h) 'v'7 SUPhEH o A(h).
19
Empirical Processes
The empirical distribution of arandom sample is the uniform discrete
measure on the observations. In this chapter, we study the convergence
of this measure and in particular the convergence of the corresponding
distribution function. This leads to laws of large numbers and central
limit theorems that are uniform in classes of functions. We also discuss a
number of applications of these resuIts.
19.1 Empirical Distribution Functions
Let XI, . . . , X n be arandom sample from a distribution function F on the real line. The
empirical distribution function is defined as
1 n
IFn(t) == - Ll{X i < t}.
n. 1
l=
It is the natural estimator for the underlying distribution F if this is completely unknown.
Because nIFn(t) is binomially distributed with mean nF(t), this estirnator is unbiased. By
the law of large numbers it is also consistent,
as
IFn(t) -+ F(t), every t.
By the centrallirnit theorem it is asymptotically normal,
v'n(JF n (t) - F(t)) -v-> N (O, F(t)( 1 - F(t))).
In this chapter we improve on these results by considering t 1---* IF n (t) as arandom function,
rather than as a real- valu ed estirnator for each t separately. This is of interest on its own
account but also provides a useful starting tool for the asymptotic analysis of other statistics,
such as quantiles, rank statistics, or trimmed means.
The Glivenko-Cantelli theorem extends the law of large numbers and gives uniform
convergence. The uniform distance
IIIFn - Flloo == supIIFn(t) - F(t)1
t
is known as the Kolmogorov-Smirnov statistic.
265
266
Empirical Processes
19.1 Theorem (Glivenko-Cantelli). If XI, X 2 , . . . are i. i.d. random variables with distri-
butionfunction F, then IIIFn - Flloo O.
as as
Proof. By the strong law of large numbers, both IF n (t) -+ F (t) and IF n (t -) -+ F (t - ) for
every t. Given a fixed £ > O, there exists a partition -00 == to < tI < . . . < tk == 00 such
that F (ti -) - F (ti -1) < £ for every i. (Point s at which F jumps more than £ are points of
the partition.) N ow, for ti -1 < t < ti,
IFn(t) - F(t) < IFn(ti-) - F(ti-) + 8,
IFn(t) - F(t) > IFn(ti-I) - F(ti-I) - £.
The convergence of IF n (t) and IF n (t - ) for every fixed t is certainly uniform for t in the finite
set {tI, ..., tk-I}. Conclude that lim sup IIIFn - Flloo < £, almost surely. This is true for
every £ > O and hence the limit superior is zero. .
The extension of the central limit theorem to a "uniform" or "functional" central limit
theorem is more involved. A first step is to prove the joint weak convergence of finitely
many coordinates. By the multivariate central limit theorem, for every t] , . . . , tk,
-Jn(IFn(ti) - F(ti),"., IFn(tk) - F(tk)) (Gp(tI),..., Gp(tk)),
where the vector on the right has a multivariate-normal distribution, with mean zero and
covarlances
EGp(ti)Gp(tj) == F(ti /\ tj) - F(ti)F(tj).
(19.2)
This suggests that the sequence of empirical processes -Jn (IF n - F), viewed as random
functions, converges in distribution to a Gaussian process G p with zero mean and covariance
functions as in the preceding display. According to an extension of Donsker' s theorem,
this is true in the sense of weak convergence of these processes in the Skorohod space
D[ -00,00] equipped with the uniform norm. The limit process G p is known as an F-
Brownian bridge process, and as a standard (or uniform) Brownian bridge if F is the
uniform distribution A on [O, 1]. From the form of the covariance function it is clear that the
F -Brownian bridge is obtainable as GA o F from a standard bridge GA' The name "bridge"
results from the fact that the sample paths of the process are zero (one says "tied down")
at the endpoints - 00 and 00. This is a consequence of the fact that the difference of two
distribution functions is zero at these points.
19.3 Theorem (Donsker). If XI, X 2 , . .. are i.i.d. random variables with distribution
function F, then the sequence of empirical processes Jfi (IF n - F) converges in distribution
in the space D[ -00,00] to a tight random element G p , whose marginal distributions are
zero-mean normal with covariance function (19.2).
Proof. The proof of this theorem is long. Because there is little to be gained by considering
the special case of cells in the real line, we deduce the theorem from a more general result
in the next section. .
Figure 19.1 shows some realizations of the uniform empirical process. The roughness
of the sample path for n == 5000 is remarkable, and typical. It is carried over onto the limit
19.1 Empirical Distribution Functions
267
'<t
o
C\J
o
o
o
C\J
9
'<t
9
0.0
0.8
1.0
0.2
0.4
0.6
co
o
tO
o
'<t
o
C\J
o
o
o
C\J
9
tO
9
0.0
0.2
1.0
0.4
0.6
0.8
'<t
o
C\J
o
o
o
C\J
9
'<t
9
tO
9
co
9
0.0
_0.2
1.0
0.4
0.6
0.8
Figure 19.1. Three realizations of the uniform empirical process, of 50 (top), 500 (middle), and 5000
(b ottom) observations, respectively.
process, for it can be shown that, for every t,
. . IGA(t+h)-GA(t)1 . IGA(t+h)-GA(t)1
O < hm Inf < hm sup < 00, a.s.
h---+O V ih log log h I - h---+O V ih log log hi
Thus, the increments of the sample paths of a standard Brownian bridge are close to being of
the order -JThT. This means that the sample paths are continuous, but nowhere differentiable.
268
Empirical Processes
A related process is the Brownian motion process, which can be defined by ZA (t)
GA (t) + t Z for a standard normal variable Z independent of GA' The addition of t Z
"liberates" the sample paths at t == 1 but retains the "tie" at t == O. The Brownian motion
process has the same modulus of continuity as the Brownian bridge and is considered an
appropriate model for the physical Brownian movement of partieles in a gas. The three
coordinates of a particle starting at the origin at time O would be taken equal to three
independent Brownian motions.
The one-dimensional empirical process and its limits have been studied extensively. t
For instance, the Glivenko-Cantelli theorem can be strengthened to a law of the iterated
logarithm,
lim sup
n ---+ 00
n
IIIFn - Flloo < ,
2log log n
a. s. ,
with equality if F takes on the value . This can be further strengthened to Strassen's
theorem
(IFn - F) H o F, a.s.
2log log n 'v'-7
Here H o F is the class of all functions h o F if h : [O, 1] 1-+ JR ranges over the set of absolutely
continuous functions t with h(O) == h(l) == O and 101 h'(s)2 ds < 1. The notation h n :: H
means that the sequence h n is relatively compact with respect to the uniform norm, with
the collection of alllirnit points being exactly equal to H. Strassen's theorem gives a fairly
precise idea of the fluctuations of the empirical process -Jii (IF n - F), when striving in law
to G F .
The preceding re suits show that the uniform distance of IFn to F is maximally of the
order ,J log log n I n as n ---+ 00. It is also known that
liminf J 2n log log n IIIFn - Flloo == Jr , a.s.
n---+oo 2
Thus the uniform distance is asymptotically (along the sequence) at least 1/(n log log n).
A famous theorem, the DKW inequality after Dvoretsky, Kiefer, and Wolfowitz, gives a
bound on the tail probabilities of IIIFn - Flloo. For every x
p( ,JnIIIF n - F 1100 > x) < 2e- 2x2 .
n
'v'-7
The originally DKW inequality did not specify the leading constant 2, which cannot be
improved. In this form the inequality was found as recently as 1990 (see [103]).
The centrailirnit theorem can be strengthened through strong approximations. These
give a special construction of the empirical process and Brownian bridges, on the same
probability space, that are close not only in a distributional sense but also in a pointwise
sense. One such result asserts that there exists a probability space carrying i.i.d. random
variables XI, X 2, . . . with law F and a sequence of Brownian bridges G F,n such that
. -Jii
hmsup 2 11,Jn(IF n - F) - GF,nll oo < 00, a.s.
n---+oo (log n)
t See [134] for the following and many other results on the univariate empirical proeess.
+ A funetion is absolutely continuous if it is the primitive funetion f g(s) ds of an integrable funetion g. Then it
is almost -everywhere differentiable with derivative g.
19.2 Empirical Distributions
269
Because, by construction, every G F,n is equal in law to G F, this implies that ,Jn (IF n -
F) G F as a process (Donsker's theorem), but it implies a lot more. Apparently, the
distance between the sequence and its limit is of the order O ( (log n)2 / ,Jn). After the
method of proof and the country of origin, results of this type are also known as Hungarian
embeddings. Another construction yields the estimate, for fixed constants a, b, and c and
every x > O,
( alog n + x )
p 11v'n(JF n - F) - GP,n 1100 > ,Jn < he-ex.
19.2 Empirical Distributions
Let XI, . . . , X n be arandom sample from a probability distribution P on a measurable space
(X, A). The empirical distribution is the discrete uniform measure on the observations. We
denote it by ITD n == n-I L7=I8 Xi' where 8x is the probability distribution that is degenerate
at x. Given a measurable function i : X JR, we write ITD n i for the expectation of i under
the empirical measure, and P i for the expectation under P. Thus
1 n
ITDni == - Li(X i ),
n. 1
l=
Pi = f i dP.
Actually, this chapter is concemed with these maps rather than with ITD n as a measure.
By the law of large numbers, the sequence ITD n i converges almost surely to P i, for every
i such that P i is defined. The abstract Glivenko-Cantelli theorems make this result uniform
in i ranging over a class of functions. A class :F of measurable functions i : X JR is
called P -Glivenko-Cantelli if
as*
IIPni - PillF == sup IIfDni - Pil -+ O.
fEF
The empirical process evaluated at i is de!ined as Gni == ,Jn(IfDni - P i). By the
multivariate central limit theorem, given any finite set of measurable functions Ji with
pi? < 00,
(G n il, . . . , G n ik) (G P il, . . . , G P ik) ,
where the vector on the right possesses a multivariate-normal distribution with mean zero
and covariances
EG p i G p g == P i g - P i P g .
The abstract Donsker theorems make this result "uniform" in classes of functions. A class
:F of measurable functions i: x JR is called P -Donsker if the sequence of processes
{Gnf: i E :F} converges in distribution to a tight limit process in the space fOO(:F). Then
the limit process is a Gaussian process G p with zero mean and covariance function as given
in the preceding display and is known as a P-Brownian bridge. Of course, the Donsker
property includes the requirement that the sample paths i Gni are uniformly bounded
for every n and every realization of XI, . . . , X n. This is the case, for instance, if the class :F
270
Empirical Processes
has a finite and integrable envelope function F: a function such that I J (x) I < F (x) < 00,
for every x and J. It is not required that the function x f---+ F (x) be uniformly bounded.
For convenience of terminology we define a class :F of vector- valu ed functions J : x f---+ ffi.k
to be Glivenko-Cantelli or Donsker if each of the classes of coordinates Ji : X f---+ IR with
J == (Ji, . . . , Jk) ranging over :F (i == 1, 2, . . . , k) is Glivenko-Cantelli or Donsker. It can
be shown that this is equivalent to the union of the k coordinate classes being Glivenko-
Cantelli or Donsker.
Whether a class of functions is Glivenko-Cantelli or Donsker depends on the "size" of
the class. A finite class of integrable functions is always Glivenko-Cantelli, and a finite
class of square- integrable functions is always Donsker. On the other hand, the class of
all square-integrable functions is Glivenko-Cantelli, or Donsker, only in trivial cases. A
relatively simple way to measure the size of a class :F is in term s of entropy. We shalI
mainly consider the bracketing entropy relative to the Lr (P)-norm
IIJIlP,r == (PIJlr)I/r.
Given two functions I and u, the bracket [l, u] is the set of all functions f with I < J < u.
An 8-bracket in Lr (P) is a bracket [I, u] with P (u - l)r < 8 r . The bracketing number
N[ ] ( £, :F, L r (P) ) is the minimum number of 8 - brackets needed to cover :F. (The bracketing
functions I and u must have finite Lr(P)-norms but need not belong to :F.) The entropy
with bracketing is the logarithm of the bracketing number.
A simple condition for a class to be P -Glivenko-Cantelli is that the bracketing numbers
in LI (P) are finite for every 8 > O. The proof is a straightforward generalization of the
proof of the classical Glivenko-Cantelli theorem, Theorem 19.1, and is omitted.
19.4 Theorem (Glivenko-Cantelli). Every class :F of measurable functions such that
N[] (8, :F, LI (P)) < 00 for every 8 > O is P-Glivenko-Cantelli.
For most classes of interest, the bracketing numbers N[] (£, :F, Lr (P)) grow to infinity
as 8 + O. A sufficient condition for a class to be Donsker is that they do not grow too fast.
The speed can be measured in terms of the bracketing integral
J[] (8, :F, L 2 (P») = 1 8 j log N[] (8, :F, L 2 (P») dIO.
If this integral is finite- valued, then the class :F is P - Donsker. The integrand in the integral
is a decreasing function of 8. Renee, the convergence of the integral depends only on the
size of the bracketing numbers for 8 + O. Because 101 8- r d8 converges for r < 1 and
diverges for r > 1, the integral condition roughly requires that the entropies grow of slower-
order than (1/ 8 ) 2 .
19.5 Theorem (Donsker). Every class:F oJmeasurablefunctions with J[] (1,:F, L 2 (P))
<00 is P-Donsker.
Proof. Let 9 be the collection of all differences J - g if J and g range over:F. With
a given set of 8-brackets [li, Ui] over :F we can construct 28-brackets over Q by tak-
ing differences [li - u j, ui - I j] of upper and lower bounds. Therefore, the bracket-
ing numbers N[] (8, Q, L 2 (P)) are bounded by the squares of the bracketing numbers
19.2 Empirical Distributions
271
N[](£/2, F, L 2 (P)). Taking a logarithm turns the square into a multiplicative factor 2,
and hence the entropy integrals of F and 9 are proportional.
For a given, s mall 8 > O choose a minimal number ofbrackets of size 8 that cover F, and
use them to form apartition of F == U i F i in sets of diameters smaller than 8. The subset of
9 consisting of differences I - g of functions I and g belonging to the same partitioning
set consists of functions of L 2 (P)-norm smaller than 8. Renee, by Lemma 19.34 ahead,
there exists a finite number a(8) such that
E*sup sup IGn(/-g)I:SJ[](8,F,L2(P))+01PF1{F>a(8)01}.
i f,gEn
Rere the envelope function F can be taken equal to the supremum of the absolute values of
the upper and lower bounds of finitely many brackets that cover F, for instance a minimal
set of brackets of size 1. This F is square- integrable.
The second term on the right is bounded by a(8)-lPF 2 1{F > a(8),Jn} and hence
converges to zero as n 00 for every fixed 8. The integral converges to zero as 8 O.
The theorem follows from Theorem 18.14, in view of Markov' s inequality. .
19.6 Example (Distributionfunction). If F is equal to the collection of all indicator
functions of the form Jt == l(-oo,t], with t ranging over JR, then the empirical process GnJt
is the classical empirical process ,Jn (IF n (t) - F (t) ). The preceding theorems reduce to
the classical theorems by Glivenko-Cantelli and Donsker. We can see this by bounding the
bracketing numbers of the set of indicator functions Jt.
Consider brackets of the form [l(-oo,ti-IJ, l(-oo,td] for a grid of points -00 == to <
tI < ... < tk == 00 with the property F (ti -) - F (ti -1) < £ for each i. These brackets
have LI (F)-size £. Their total number k can be chosen smaller than 2/£. Because F J2 <
F J for every O < J < 1, the L 2 (F)-size of the brackets is bounded by -Je. Thus
N[](-Je, F, L 2 (F)) < (2/£), whence the bracketing numbers are of the polynomial order
(1/£)2. This means that this class of functions is very small, because a function of the type
10g(1/ £) satisfies the entropy condition of Theorem 19.5 easily. D
19.7 Example (Parametric class). Let F == {Je: e E 8} be a collection of measurable
functions indexed by a bounded subset 8 c JRd. Suppose that there exists a measurable
function m such that
I leI (x) - le 2 (x)/ < m(x)l!e l - e 2 1!, every el, e 2 .
If P I mir < 00, then there exists a constant K, depending on 8 and d only, such that the
bracketing numbers satisfy
( diam 8 ) d
N[](6'lI m llp,r,;::, Lr(P)) < K 6' '
every O < £ < diam 8.
Thus the entropy is of smaller order than log( 1 / £). Renee the bracketing entropy integral
certainly converges, and the class of functions Fis Donsker.
To establish the upper bound we use brackets of the type [Je - £m, Je + £m] for e
ranging over a suitably chosen subset of 8. These brackets have Lr(P)-size 2£llmllp,r. If
e ranges over a grid of meshwidth £ over 8, then the brackets cover F, because by the
Lipschitz condition, JeI - £m < J e 2 < leI + £m if II el - e 2 11 < £. Thus, we ne ed as many
brackets as we need balls of radius £ /2 to cover 8.
272
Empirical Processes
The size of 8 in every fixed dimension is at most diam 8. We can cover 8 with fewer
than (diam 8 / E)d cubes of size E. The circumscribed balls have radius a multiple of 8 and
also cover 8. If we replace the centers of these balls by their projections into 8, then the
balls of twice the radius still cover 8. D
19.8 Example (Pointwise Compact Class). The parametric class in Example 19.7 is cer-
tainly Glivenko-Cantelli, but for this a much weaker continuity condition also suffices. Let
F == {Je: e E 8} be a collection of measurable functions with integrable envelope function
F indexed by a compact metric space 8 such that the map e I---)- Je (x) is continuous for
every x. Then the L 1 -bracketing numbers of F are finite and hence Fis Glivenko-Cantelli.
We can construct the brackets in the obvious way in the form [JB, JB], where B is an
open ball and JB and JB are the infimum and supremum of Je for e E B, respectively.
Given a sequence of balls Bm with common center a given e and radii decreasing to O, we
have JBm - JB m + Je - Je == O by the continuity, pointwise in x and hence also in LI
by the dominated-convergence theorem and the integrability of the envelope. Thus, given
E > O, for every e there exists an open ball B around e such that the bracket [J B, f B] has
size at most E. By the compactness of 8, the collection of balls constructed in this way has
a finite subcover. The corresponding brackets cover F.
This construction shows that the bracketing numbers are finite, but it gives no control on
their sizes. D
19.9 Example (Smooth functions). Let JRd == U j Ij be a partition in cubes of volu me 1
and let F be the class of all functions J : JRd ---+ JR whose partial derivatives up to order cl
exist and are uniformly bounded by constants Mj on each of the cubes Ij. (The condition
includes bounds on the "zero-th derivative," which is f itself.) Then the bracketing numbers
of F satisfy, for every V > d / cl and every probability measure P,
( 1 ) v ( 00 V ) V:r
logN[](.s,F,Lr(P») < K; (MjP(Ij») V+' .
The constant K depends on cl, V, r, and d only. If the series on the right converges for r == 2
and some dj cl < V < 2, then the bracketing entropy integral of the class F converges and
hence the class is P - Donsker. t This requires sufficient smoothness cl > d /2 and sufficiently
small tail probabilities P (I j ) relative to the uniform bounds Mj. If the functions f have
compact support (equivalently Mj == O for al1large j), then smoothness of order cl > dj2
suffices. D
19.10 Example (Sobolev classes). Let F be the set of all functions J : [O, 1] I---)- JR such
that II f 1100 < 1 and the (k -1)-th derivative is absolutely continuous with f (J(k»)2(x) dx <
1 for some fixed kEN. Then there exists a constant K such that, for every 8 > 0,+
( 1 ) lik
log N[](.s, F, II . 1100) < K; .
Thus, the class F is Donsker for every k > 1 and every P. D
t The upper bound and this sufficient eondition can be slightly improved. For this and a proof of the upper bound,
see e.g., [146, Corollary 2.74].
:j: See [16] .
19.2 Empirical Distributions
273
19.11 Example (Bounded variation). Let:F be the collection of all monotone functions
f : JR. r--+ [ -1, 1], or, bigger, the set of all functions that are of variation bounded by 1. These
are the differences of pairs of monotonely increasing functions that together increase at most
1. Then there exists a constant K such that, for every r > 1 and probability measure P, t
logN[](8,F,L 2 (P)) < K( ).
Thus, this class of functions is P-Donsker for every P. O
19.12 Example (Weighted distributionfunction). Let w : (O, 1) r--+ IR+ be a fixed, con-
tinuous function. The weighted empirical process of a sample of real-valued observations
is the process
t r--+ er (t) == (lF n - F) (t ) w ( F (t) )
(defined to be zero if F (t) == O or F (t) == 1). For a bounded function w, the map z r--+ z. W o F
is continuous from £00[-00,00] into f oo [ -00,00] and hence the weak convergence of
the weighted empirical process follows from the convergence of the ordinary empirical
process and the continuous- mapping theorem. Of more interest are weight functions that
are unbounded at O or 1, which can be used to rescale the empirical process at its two
extremes -00 and 00. Because the difference (lF n - F) (t) converges to O as t ---+ ::too,
the sample paths of the process t r--+ er (t) may be bounded even for unbounded w, and the
rescaling increases our know ledge of the behavior at the two extremes.
A simple condition for the weak convergence of the weighted empirical process in
£00 ( -00, (0) is that the weight function w is monotone around O and 1 and satisfies
101 w 2 (s) ds < 00. The square-i ntegrabi lity is almost necessary, because the convergence
is known to fail for w(t) == 1/ ,J t(l - t). The Chibisov-O'Reilly theorem gives necessary
and sufficient conditions but is more complicated.
We shalI give the proof for the case that w is unbounded at only one endpoint and
decreases from w(O) == 00 to w(l) == O. Furthermore, we assume that F is the uni-
form measure on [O, 1]. (The general case can be treated in the same way, or by the
quantile transformation.) Then the function ves) == w 2 (s) with domain [O, 1] has an
inverse v-I(t) == w-I(,Ji) with domain [0,00]. A picture of the graphs shows that
Jo oo w-I(,Ji) dt == Jo I w 2 (t) dt, which is finite by assumption. Thus, given an 8 > O,
we can choose partitions O == So < Sl < ... < Sk == 1 and O == to < tI < . .. < tz == 00
such that, for every i,
[Si w 2 (s) ds < 8 2 ,
Si-l
iti w- 1 (,Ji) dt < 8 2 ,
ti-l
This corresponds to slicing the area under w 2 both horizontally and vertically in pieces of
size 8 2 . Let the partition O == Uo < Ul < ... < um == 1 be the partition consisting of all
points S i and all points w -1 ( ,Ji j ). Then, for every i,
1 W 2 (Ui_l)
(W 2 (Ui_1) - W 2 (Ui))Ui_1 < w- 1 (,Jt)dt < 8 2 .
W 2 (Ui)
t See, e.g., [146, Theorem 2.75J.
274
Empirical Processes
It follows that the brackets
[w 2 (ui)1[o,ui_rJ, W 2 (Ui-I)1[o,ui_rJ + w 2 1[(ui_l,ud]
have LI (A)-size 28 2 . Their square roots are brackets for the functions of interest x 1---+ w(t)
1 [O,t] (x), andhave L 2 (A)-size ,)28, because PI-JU - 01 2 < Plu -II. Because the number
m of points in the partitions can be chosen of the order (1/8)2 for small 8, the bracketing
integral of the class of functions x 1---+ W (t) 1 [O,t] (x) converges easily. D
The conditions given by the preceding theorems are not necessary, but the theorems
cover many examples. Simple necessary and sufficient conditions are not known and may
not exist. An alternative set of relatively simple conditions is based on "uniform covering
numbers." The covering number N (8, F, L 2 (Q)) is the minimal number of L 2 (Q)-balls of
radius 8 needed to cover the set F. The entropy is the logarithm of the covering number. The
following theorems show that the bracketing numbers in the preceding Glivenko-Cantelli
and Donsker theorems can be replaced by the uniform covering numbers
supN(81I F IIQ,r, F, Lr(Q)).
Q
Here the supremum is taken over all probability measures Q for which the class F is not
identically zero (and hence IIFIIQ,r == QF r > O). The uniform covering numbers are
relative to a given envelope function F. This is fortunate, because the covering numbers
under different measures Q typically are more stable if standardized by the norm II F II Q,r of
the envelope function. In comparison, in the case ofbracketing numbers we consider asingle
distribution P, and standardization by an envelope does not make much of a difference.
The uniform entropy integral is defined as
J(8, F, L 2 ) = 1
log sup N(811 F II Q,2, F, L 2 (Q)) d8.
Q
19.13 Theorem (Glivenko-Cantelli). Let F be a suitably measurable class of measurable
functionswithsuPQN(81IFIIQ,I,F,LI(Q)) < ooforevery8 > O. IfP*F < 00, thenF
is P -Glivenko-Cantelli.
19.14 Theorem (Donsker). Let F be a suitably measurable class ofmeasurablefunctions
with J (1, F, L 2 ) < 00. If P* F 2 < 00, then F is P -Donsker.
The condition that the class F be "suitably measurable" is satisfied in most examples
but cannot be omitted. We do not give a general definition here but note that it suffices that
there exists a countable collection Q of functions such that each f is the pointwise limit of
a sequence gm in Q.t
An important class of examples for which good estimates on the uniform covering
numbers are known are the so-called Vapnik-Červonenkis classes, or VC classes, which are
defined through combinatorial properties and include many well-known examples.
t See, for example, [117], [120], or [146] for proofs of the preeeding theorems and other unproven results in this
seetion.
19.2 Empirical Distributions
275
t
Figure 19.2. The subgraph of a function.
Say that a collection C of subsets of the sample space X picks out a certain subset A of
the finite set {XI, . . . , Xn} C Xif it can be written as A == {XI, . . . , Xn} n C for some C E C.
The collection C is said to shatter {XI, . . . , Xn} if C picks out each of its 2 n subsets. The VC
index V (C) of C is the smallest n for which no set of size n is shattered by C. A collection
C of measurable sets is called a VC class if its index V (C) is finite.
More generally, we can de fine VC classes of functions. A collection F is a VC class of
functions if the collection of all subgraphs { (x, t) : f (x) < t }, if f ranges over F, forms a
VC class of sets in X x IR (Figure 19.2). It is not difficult to see that a collection of sets C
is a VC class of sets if and only if the collection of corresponding indicator functions le is
a VC class of functions. Thus, it suffices to consider VC classes of functions.
By definition, a VC class of sets picks out strictly less than 2 n subsets from any set of
n > V (C) elements. The surprising fact, known as Sauer' s lemma, is that such a class can
necessarily pick out only a polynomial number O (n V(C)-l) of subsets, well below the 2 n -1
that the definition appears to allow. N ow, the number of subsets picked out by a collection
C is closely related to the covering numbers of the class of indicator functions {1 e : C E C}
in LI (Q) for discrete, empirical type measures Q. By a clever argument, Sauer' s lemma
can be used to bound the uniform covering (or entropy) numbers for this class.
19.15 Lemma. There exists a universal constant K such that for any vc class F of
functions, any r > 1 and O < £ < 1,
( 1 ) r(V(F)-l)
S PN(811FIIQ,,,F,Lr(Q)) < KV(F)(16e)V(F) '
Consequently, VC classes are examples of polynomial classes in the sense that their
covering numbers are bounded by a polynomial in 1/£. They are relatively small. The
276
Enlpirical Processes
upper bound shows that VC classes satisfy the entropy conditions for the Glivenko-Cantelli
theorem and Donsker theorem discussed previously (with much to spare). Thus, they are P-
Glivenko-Cantelli and P - Donsker under the moment conditions P * F < 00 and P * p2 < 00
on the ir envelope function, if they are "suitably measurable." (The VC property does not
imply the measurability.)
19.16 Example (Cells). The collection of all cells (-00, tJ in the real line is a VC class
of index V (C) == 2. This follows, because every one-point set {XI} is shattered, but no
two-point set {XI, X2} is shattered: If XI < X2, then the cells (-00, tJ cannot pick out
{X2}. D
19.17 Example (Vector spaces). Let:F be the set of alllinear combinations L Ai Ji of a
given, finite set of functions il, . . . , ik on X. Then:F is a VC class and hence has a finite
uniform entropy integral. Furthermore, the same is true for the class of all sets {I > c} if
i ranges over 1 and c over ffi..
For instance, we can construct :F to be the set of all polynomials of degree less than
some number, by taking basis functions 1, X, x2, . . . on JR and functions Xfl . . . X d more
generall y. For pol ynomials of degree up to 2 the collection of sets {i > O} contains
already all half-spaces and all ellipsoids. Thus, for instance, the collection of all ellipsoids
is Glivenko-Cantelli and Donsker for any P.
To prove that :F is a VC class, consider any collection of n == k + 2 points (XI, tI), . . . ,
(Xn, t n ) in Xx ffi.. We shalI show this set is not shattered by :F, whence V (:F) < n.
By assumption, the vectors (I (XI) - tI, . . . , 1 (Xn) - tn))T are contained in a (k + 1)-
dimensional subspace of ffi. n . Any vector a that is orthogonal to this subspace satisfies
L ai(/(xi) - ti) == L (-ai)(/(Xi) - ti)'
i : ai > O i : ai < O
(Define a sum over the empty set to be zero.) There exists a vec tor a with at least one
strictly positive coordinate. Then the set { (Xi, ti) : ai > O} is nonempty and is not picked
out by the subgraphs of:F. If it were, then it would be of the form { (Xi, ti) : ti < i (ti) }
for some I, but then the left side of the display would be strictly positive and the right side
nonpositive. O
A number of operations allow to build new VC classes or Donsker classes out ofknown
VC classes or Donsker classes.
19.18 Example (Stability properties). The class of all complements ce, all intersections
enD, all unions C UD, and all Cartesian products C x D of sets C and D that range over
VC classes C and V is VC.
The class of all suprema 1 v g and infima 1 /\ g of functions 1 and g that range over
VC classes :F and 9 is VC.
The proof that the collection of all intersections is VC is easy upon using Sauer' s lernma,
according to which a VC class can pick out only a polynomial number of subsets. From
n given points C can pick out at most O (n V(C)) subsets. From each of these subsets V
can pick out at most O (n V(V)) further subsets. A subset picked out by enD is equal to
the subset picked out by C intersected with D. Thus we get all subsets by following the
19.3 Goodness-of-Fit Statistics
277
two-step procedure and hence C n V can pi ck out at most O(n V(C)+V(V)) subsets. For large
n this is well below 2 n , whence C n V cannot pick out all subsets.
That the set of all complements is VC is an immediate consequence of the definition.
Next the result for the unions follows by combination, because C U D == (CC n DC)c.
The results for functions are consequences of the results for sets, because the subgraphs
of suprema and infima are the intersections and unions of the subgraphs, respectively. D
19.19 Example (Uniform entropy). If F and Q possess a finite uniform entropy inte-
gral, relative to envelope functions F and G, then so does the class FQ of all functions
x 1---+ f (x) g (x), relative to the envelope function F G .
More generally, suppose that 1> : JR2 1---+ JR is a function such that, for given functions L f
and Lg and every x,
14>(11 (x), gj (x)) - 4>(12(X), g2(X)) I < L f(x)l!I - !zl(x) + Lg(x)lgj - g21(x).
Then the class of all functions 1> (f, g) - 1> (fo, go) has a finite uniform entropy integral
relative to the envelope function L fF + Lg G, whenever F and Q have finite uniform entropy
integrals relative to the envelopes F and G. D
19.20 Example (Lipschitz transformations). For any fixed Lipschitz function 1> : JR2 1---+ JR,
the class of all functions of the form 1> (f, g) is Donsker, if f and g range over Donsker
classes F and Q with integrable envelope functions.
For example, the class of all sums f + g, all minima f 1\ g, and all maxima f v g are
Donsker. If the classes F and Q are uniformly bounded, then also the products f g form
a Donsker class, and if the functions f are uniformly bounded away from z ro, then the
functions 1/ f form a Donsker class. D
19.3 Goodness-of-Fit Statistics
An important application of the empirical distribution is the testing of goodness-of-fit.
Because the empirical distribution ]p n is always a reasonable estimator for the underlying
distribution P of the observations, any measure of the discrepancy between ]p n and P can
be used as a test statistic for testing the hypothesis that the true underlying distribution is
P.
Some popular global measures of discrepancy for real-valued observations are
JnllIF n - Flloo, (Kolmogorov-Smirnov),
n f (JF n - F)2 dF, (Cramer-von Mises).
These statistics, as well as many others, are continuous functions of the empirical process.
The continuous-mapping theorem and Theorem 19.3 immediately imply the following
result.
19.21 Corollary. If XI, X 2 , . . . are i.i.d. randoln variables with distribution function F,
then the sequences of Kolmogorov-Smirnov statistics and Cramer-von Mises statistics con-
verge in distribution to IIG F 1100 and J G} d F, respectively. The distributions ofthese limits
are the same for every continuous distribution function F.
278
Empirical Processes
Proof. The maps z 1-* Ilzlloo and z 1-* J Z2(t) dt from D[ -00,00] into IR are continuous
with respect to the supremum norm. Consequently, the first assertion folIows from the
continuous- mapping theorem. The second assertion folIows by the change of variables
F (t) 1-* U in the representation G F == GA o F of the Brownian bridge. Altematively, use
the quantile transformation to see that the Kolmogorov-Smirnov and Cramer-von Mises
statistics are distribution - free for every fixed n. .
It is probably practicalIy more relevant to test the goodness-of-fit of compositive nulI
hypotheses, for instance the hypothesis that the underlying distribution P of arandom
sample is normal, that is, it belongs to the normallocation-scale family. To test the nulI
hypothesis that P belongs to a certain family {P() : () E 8}, it is natural to use a measure
of the discrepancy between P n and Pe, for a reasonable estimator e of (). For instance, a
modified Kolmogorov-Smirnov statistic for testing normality is
( t - X )
s p vn 1Fn(t) - <I> S .
For many goodness-of- fit statistics of this type, the limit distribution folIows from the limit
distribution of vn (P n - P e ). This is not a Brownian bridge but also contains a "drift," due
to e. InformalIy, if () 1-* P() has a derivative P() in an appropriate sense, then
vn(Pn - Pe) == vn(Pn - P()) - vn(Pe - p()),
vn(Pn - P()) - vn(e - ())T P().
(19.22)
By the continuous-mapping theorem, the limit distribution of the last approximation can be
derived from the limit distribution of the sequence vn (P n - p(), e - ()). The first component
converges in distribution to a Brownian bridge. Its joint behavior with vn (e - ()) can most
easily be obtained if the latter sequence is asymptoticalIy linear. Assume that
A 1 n
vn(()n - ()) = vn Vro (Xi) + O P. (1),
for "influence functions" 1/J() with P() 1/J() == O and P() 111/J() 11 2 < 00.
19.23 Theorem. Let XI, . . . , X n be arandom sample from a distribution P() indexed by
() E IR k . Let F be a P()-Donsker class ofmeasurablefunctions and let en be estimators that
are asymptotically linear with influence function 1/J(). Assume that the map () 1-* P() from IR k
to goo (F) is Frechet differentiable at (). t Then the sequence vn (P n - Pe) converges under
() in distribution in goo (F) to the process f 1-* G Pe f - G Pe 1/Jl P() f.
Proof. In view of the differentiability of the map () 1-* P() and Lemma 2.12,
II Pen - p () - (e - ()) T P () II F == o p ( II en - e II) .
This justifies the approximation (19.22). The class 9 obtained by adding the k components
of 1/J() to F is Donsker. (The union of two Donsker classes is Donsker, in general. In
t This means that there exists a map Pe : F f--+ IR k such that II Pe+h - Pe - h T Pe IIF = o(llh II) as h --+ O; see
Chapter 20.
19.4 Random Functions
279
the present case, the result also follows directly from Theorem 18.14.) The variables
(-Jii (]P n - P()), n -1/2 L 1j;() (Xi) ) are obtained from the empirical process seen as an element
of £00 (9) by a continuous map. Finally, apply Slutsky's lemma. .
The preceding theorem implies, for instance, that the sequences of modified Kolmogorov-
Smimov statistic -JiiIIF n - Fe 1100 converge in distribution to the supremum of a certain
Gaussian process. The distribution of the limit may depend on the model e 1---+ F(), the
estimators en, and even on the parameter value e. Typically, this distribution is not known
in closed form but has to be approximated numerically or by simulation. On the other
hand, the limit distribution of the true Kolmogorov-Smimov statistic under a continuous
distribution can be derived from properties of the Brownian bridge, and is given by t
00
p(IIGAlloo > x) == 2 L(_1)J+l e -2 J2 x 2 .
J=1
With the Donsker theorem in hand, the route via the Brownian bridge is probably the
most convenient. In the 1940s Smirnov obtained the right side as the limit of an explicit
expression for the distribution function of the Kolmogorov-Smirnov statistic.
19.4 Random Functions
The language of Glivenko-Cantelli classes, Donsker classes, and entropy appears to be
convenient to state the "regularity conditions" needed in the asymptotic analysis of many
statistical procedures. For instance, in the analysis of Z- and M -estimators, the theory
of empirical processes is a powerful tool to control remainder terms. In this section we
consider the key element in this application: controlling random sequences of the form
2:::7=1 fn,e n (Xi) for functions In,() that change with n and depend on an estimated parameter.
If a class :F of functions is P -Glivenko-Cantelli, then the difference I JP ni - Pil converges
to zero uniformly in 1 varying over :F, almost surely. Then it is immediate that also
A A as A
IJP n 1 n - p Ini ---+ O for every sequence of random functions In that are contained in:F. If
In converges almost surely to a function 10 and the sequence is dominated (or uniformly
A A
integrable), so that P In ---+ Pio, then it follows that JP n l n ---+ P Ja.
Here by "random functions" we mean measurable functions x 1---+ In (x; w) that, for
every fixed x, are real maps defined on the same probability space as the observations
X 1 (w),..., Xn(w). Inmany examples the function In(x) == In(x; XI,..., X n ) is afunc-
tion of the observations, for every fixed x. The notations JP n f n and P I n are abbreviations
for the expectations of the functions x 1---+ In (x; w) with w fixed.
A similar principle applies to Donsker classes of functions. For a Donsker class :F, the
empirical process CG n l converges in distribution to a P-Brownian bridge process CG pI "uni-
formly in 1 E :F." In view of Lemma 18.15, the limiting process has uniform1y continuous
sample paths with respect to the variance semimetric. The uniform convergence combined
with the continuity yields the weak convergence CG n I n CG p 10 for every sequence I n of
random functions that are contained in :F and that converges in the variance semimetric to
a function Ja.
t See, for instance, [42, Chapter 12], or [134].
280
En1pirical Processes
19.24 Lemma. Suppose that Fis a P-Donsker class ofmeasurablefunctions and In is a
sequence oj random functions that take their values in :F such that J (I n (x) - Ja (x ) ) 2 d P (x)
converges in probability to O for some fo E L 2 (P). Then Gn(ln - fo) O and hence
Gnl n Gpfo.
Proof. Assume without of loss of generality that fo is contained in F. Define a function
g: fOO(:F) x F'r--+ JR by g(z, f) == zef) - z(fo). The set:F is a semimetric space relative
to the L 2 (P)-metric. The function g is continuous with respect to the product semimetric
at every point (z, f) such that f'r--+ zef) is continuous. Indeed, if (Zn, fn) --+ (z, f) in the
space fOO(F) x F, then zn --+ z uniformly and hence zn(fn) == Z(Jn) + 0(1) --+ zef) if z
is continuous at f.
By assumption, In fo as maps in the metric space F. Because F is Donsker,
G n G p in the space fOO(F), and it follows that (G n , In) (CG p , fo) in the space
foo (F) x F. By Lemma 18.15, almost all sample paths of G p are continuous on F. Thus
the function g is continuous at almost every point (CG p, fo). By the continuous-mapping
theorem, G n (I n - fo) == g (G n , IrJ g (G p, fo) == O. The lemma follows, because
convergence in distribution and convergence in probability are the same for adegenerate
limit. .
The preceding lemma can also be proved by reference to an almost sure representation
for the converging sequence G n G p. Such a representation, a generalization of Theorem
2.19 exists. However, the correct handling of measurability issues makes its application
invol ved.
19.25 Example (Mean absolute deviation). The mean absolute deviation of arandom
sample XI, . . . , X n is the scale estimator
1 n _
M n == - LIX i - Xnl.
n i=l
The absolute value bars make the derivation of its asymptotic distribution surprisingly
difficult. (Try and do it by elementary means.) Denote the distribution function of the
observations by F, and assume for simplicity of notation that they have mean F x equal to
zero. We shalI write JFnlx - el for the stochastic process e 'r--+ n-1I:7=1IXi - el, and use
the notations CG n Ix - e I and F Ix - e I in a similar way.
If F x 2 < 00, then the set of functions x 'r--+ Ix - e I with e ranging over a compact, such
as [-1, 1], is F -Donsker by Example 19.7. Because, by the triangle inequality, F(lx -
X nl - I x l)2 < I X nl 2 O, the preceding lemma shows that Gnlx - X nl - Gnlxl . O.
This can be rewritten as
(Mn - Flxl) == (Flx - X nl - Flxl) + CGnlxl + op(l).
If the map e 'r--+ F Ix - e I is differentiable at O, then, with the derivative written in the form
2F (O) - 1, the first term on the right is asymptotically equivalent to (2F (O) - 1) Gnx, by
the delta method. Thus, the mean absolute deviation is asymptotically normal with mean
zero and asymptotic variance equal to the variance of (2F (O) - 1) XI + IX 11.
If the mean and median of the observations are equal (i.e., F(O) == ), then the first term
is O and hence the centering of the absolute values at the sample mean has the same effect
19.4 Random Functions
281
as centering at the true mean. In this case not knowing the true mean does not hurt the scale
estimator. In comparison, for the sample variance this is true for any F. O
Perhaps the most important application of the preceding lemma is to the theory of Z-
estimators. In Theorem 5.21 we imposed a pointwise Lipschitz condition on the maps
8 1---+ 1/Je to ensure the convergence 5.22:
CG n (1(; en - 1/1e o ) O.
In view of Example 19.7, this is now seen to be a consequence of the preceding lemma. The
display is valid if the class of functions { 1/1e : II 8 - 8 0 II < 8} is Donsker for some 8 > O and
1/1e -+ 1/1e o in quadratic mean. Imposing a Lipschitz condition is just one method to ensure
these conditions, and hence Theorem 5.21 can be extended considerably. In particular, in
its generalized form the theorem covers the sample median, corresponding to the choice
1/1e (x) == sign(x - 8). The sign functions can be bracketed just as the indicator functions
of cells considered in Example 19.6 and thus form a Donsker class.
For the treatment of semiparametric model s (see Chapter 25), it is useful to extend the
results on Z-estimators to the case of intinite-dimensional parameters. A differentiability
or Lipschitz condition on the maps 8 1---+ 1/1e would preciude most applications of interest.
However, if we use the language of Donsker classes, the extension is straightforward and
useful.
If the parameter 8 ranges over a subset of an intinite-dimensional normed space, then we
use an intinite number of estimating equations, which we label by some set H and assume
to be sums. Thus the estimator en (nearly) solves an equation P n 1/1e,h == O for every h EH.
We assume that, for every fixedx and 8, the map h 1---+ 1/1e,h(X), which we denote by 1/1e(x),
is uniformly bounded, and the same for the map h 1---+ P1/1e,h, which we den ote by P1/1e.
19.26 Theorem. For each 8 in a subset 8 of a normed space and every h in an arbitrary set
H, let x 1---+ 1/1e,h(X) be a measurablefunction such that the class {1/1e,h: 118 - 8 0 11 < 8, h E
H} is P -Donsker for some 8 > O, with finite envelope function. Assume that, as a map
into foo (H), the map 8 1---+ P 1/1e is Frechet-differentiable at a zero 8 0 , with a derivative
V : lin 8 1---+ ,eoo (H) that has a continuous inverse on its range. Furthermore, assume that
II p (1/Je,h - 1/1e o ,h)211 H O as e e o . lf IIP n 1/1eJI H == O P (n- 1 / 2 ) and en e o , then
V (en - 8 0 ) == -G n 1/1e o + op(l).
Proof. This follows the same lines as the proof of Theorem 5.21. The only novel aspect
is that a uniform version of Lemma 19.24 is needed to ensure that CG n (1/1 en - 1/1 e o) converges
to zero in probability in fOO(H). This is proved along the same lines.
Assume without loss of generality that en takes its values in 8 8 == {8 E 8 : II e - 8 0 II < 8}
and detine a map g: ,e00(88 x H) X 8 8 1---+ ,eOO(H) by g(z, e)h == z(e, h) - z(80, h). This
map is continuous at every point (z, e o ) such that Ilz(e, h) - z(e o , h) IIH O as e -+ e o .
The sequence (cG n 1/1e, en) converges in distribution in the space ,eoo (8 8 X H) X 8 8 to a pair
(CG1jfe, e o ). As e e o , we have that sUPh P(1/1e,h - 1/1e o ,h)2 O by assumption, and thus
II Go/ e - CG1/1e o II H -+ O almost surely, by the uniform continuity of the sample paths of the
Brownian bridge. Thus, we can apply the continuous-mapping theorem and conclude that
g(CG n te, en) 'v'7 g(cG1/1e, 8 0 ) == O, which is the desired result. .
282
Empirical Processes
19.5 Changing Classes
The Glivenko-Cantelli and Donsker theorems concem the empirical process for different
n, but each time with the same indexing class F. This is sufficient for a large number of
applications, but in other cases it may be necessary to allow the class F to change with
n. For instance, the range of the random function in in Lemma 19.24 might be different
for every n. We encounter one such a situation in the treatment of M -estimators and the
likelihood ratio statistic in Chapters 5 and 16, in which the random functions of interest
,Jn (me n - meo) - ,Jn (en - eo)me o are obtained by rescaling a given class of functions.
It tums out that the convergence of random variables such as G n i n does not require the
ranges Fn of the functions i n to be constant but depends only on the sizes of the ranges
to stabilize. The nature of the functions inside the classes could change completely from
n to n (apart from a Lindeberg condition).
Directly or indirectly, all the results in this chapter are based on the maximal inequalities
obtained in section 19.6. The most general results can be obtained by applying these
inequalities, which are valid for every fixed n, directly. The conditions for convergence
of quantities such as Gnl n are then framed in terms of (random) entropy numbers. In
this section we give an intermediate treatment, starting with an extension of the Donsker
theorems, Theorems 19.5 and 19.14, to the weak convergence of the empirical process
indexed by classes that change with n.
Let Fn be a sequence of classes of measurable functions fn,t : XI---+ ffi. indexed by a
parameter t, which belongs to a common index set T. Then we can consider the weak
convergence of the stochastic processes t 1---+ Gnfn,t as elements of gOO(T), assuming that
the sample paths are bounded. By Theorem 18.14 weak convergence is equivalent to
marginal convergence and asymptotic tightness. The marginal convergence to a Gaussian
process follows under the conditions of the Lindeberg theorem, Proposition 2.27. Sufficient
conditions for tightness can be given in terms of the entropies of the classes Fn.
We shalI assume that there exists a semimetric p that makes T into a totally bounded
space and that relates to the L 2 - metric in that
sup P (fn,s - fn,t)2 O, every 8n {. O.
p(s,t)<8 n
(19.27)
Furthermore, we suppose that the classes Fn possess envelope functions Fn that satisfy the
Lindeberg condition
PF; == 0(1),
P F;{Fn > E',Jn} O, every E' > O.
Then the central limit theorem holds under an entropy condition. As before, we can use
either bracketing or uniform entropy.
19.28 Theorem. Let Fn == {fn,t : t E T} be a class of measurable functions indexed by
a totally bounded semimetric space (T, p) satisfying (19.27) and with envelope function
that satisfies the Lindeberg condition. If J[] (8n, Fn, L 2 (P)) O for every 8n {. O, or
alternatively, every :F,1 is suitably measurable and J (8n, Fn, L 2 ) O for every 8n {. O,
then the sequence {CG n h1,t : t E T} converges in distribution to a tight Gaussian process,
provided the sequence of covariance functions P fn,s fn,t - P fn,s P fn,t converges pointwise
on T x T.
19.5 Changing Classes
283
Proof. U nder bracketing the proof of the following theorem is similar to the proof of
Theorem 19.5. We omit the proof under uniform entropy.
For every given 8 > O we can use the semimetric p and condition (19.27) to p arti ti on T
into finitely many sets TI, . . . , Tk such that, for every sufficiently large n,
sup sup P (in,s - in,t)2 < 8 2 .
ž s,tET;
(This is the only role for the totally bounded semimetric p; altematively, we could assume
the existence ofpartitions as in this display directly.) Next we apply Lemma 19.34 to obtain
the bound
PF;I{Fn > an (8),Jn}
Esup sup IGn(in,s - in,t)l;S J[](8, Fn, L 2 (P)) + .
ž s,tET; an (8)
Here an (8) is the number given in Lemma 19.34 evaluated for the class offunctions Fn - Fn
and Fn is its envelope, but the corresponding number and envelope of the class Fn differ
only by constants. Because J[] (8n, Fn, L 2 (P)) ---+ O for every 8n O, we must have that
J[](8, Fn, L 2 (P)) == 0(1) for every 8 > O and hence a n (8) is bounded away from zero.
Then the second term in the preceding display converges to zero for every fixed 8 > O,
by the Lindeberg condition. The first term can be made arbitrarily small as n ---+ 00 by
choosing 8 small, by assumption. .
19.29 Example (Local empirical measure). Consider the functions in,t == r n 1 (a,a+tD n ]
for t ranging over a compact in JR, say [O, 1], a fixed number a, and sequences 8n O and
r n ---+ 00. This leads to a multiple of the loeal empirieal measure JPnin,t == (l/n)#(X ž E
(a, a + t 8n]), which counts the fraction of observations falling into the shrinking interval s
(a, a + t8n].
Assume that the distribution of the observations is continuous with density p. Then
Pin t == r;P(a, a + t8n] == r p(a)t8n + o(r;8n).
Thus, we obtain an interesting limit only if r; 8n 1. From now on, set r; 8n == 1. Then
the variance of every G n in,t converges to a nonzero limit. Because the envelope function
is Fn == in,l, the Lindeberg condition reduces to r; P(a, a + 8n]lr n >8,Jn ---+ O, which is
true provided n8n ---+ 00. This requires that we do not localize too much. If the intervals
become too small, then catching an observation becomes a rare event and the problem is
not within the domain of normal convergence.
The bracketing numbers of the cells 1 (a,a+tD n ] with t E [O, 1] are of the order O (8n /8 2 ).
Multiplication with r n changes this in 0(1/8 2 ). Thus Theorem 19.28 applies easily, and we
conclude that the sequence of processes t r-+ G n in,t converges in distribution to a Gaussian
process for every 8n O such that n 8n ---+ 00.
The limit process is not a Brownian bridge, but a Brownian motion process, as follows
by computing the limit covariance of (G n in,s, G n in,t). Asymptotically the local empirical
process "does not know" that it is tied down at its extremes. In fact, it is an interesting
exercise to check that two different local empirical processes (fixed at two different numbers
a and b) converge jointly to two independent Brownian motions. O
In the treatment of M -estimators and the likelihood ratio statistic in Chapters 5 and 16,
we encountered random functions resulting from rescaling a given class of functions. Given
284
Empirical Processes
functions X f---+ me (x) indexed by a Euclidean parameter e, we needed conditions that ensure
that, for a given sequence r n -+ 00 and any random sequence h n == O; (1),
( -T. ) p
G n r n (meo+Jin/r n - meo) - h n meo -+ O.
(19.30)
We shaH prove this under a Lipschitz condition, but it should be clear from the following
proof and the preceding theorem that there are other possibilities.
19.31 Lemma. For each e in an open subset of Euclidean space let x f---+ me (x) be a
measurable function such that the map e f---+ me (x) is differentiable at eafor almost every x
(or in probability) with derivative meo (x) and such that, for every el and e 2 in a neighborhood
ofeo, andfor a measurable function m such that Pm 2 < 00,
Ilme l (x) - me2 (x) II < m(x) lIe l - e 2 11.
Then (19.30) is valid for every random sequence h n that is bounded in probability.
Proof. The random variables G n (rn (meo+h/rn - meo) - h T meo) have mean zero and their
variance converges to O, by the differentiability of the maps e f---+ me and the Lipschitz con-
dition, which allows application of the dominated-convergence theorem. In other words,
this sequence seen as stochastic processes indexed by h converges marginally in distribu-
tion to zero. Because the sequence h n is bounded in probability, it suffices to strengthen
this to uniform convergence in Ilh II < 1. This follows if the sequence of processes con-
verges weakly in the space £00 (h : II hil < 1), because taking a supremum is a continuous
operation and, by the marginal convergence, the weak limit is then necessarily zero. By
Theorem 18.14, we can confine ourselves to proving asymptotic tightness (i.e., condition
(ii) of this theorem). Because the linear processes h f---+ h T Gnmeo are trivially tight, we may
concentrate on the processes h f---+ G n (rn (meo+h/rn - meo)), the empirical process indexed
by the classes offunctions rnM I / rn , for Mo == {me - meo: Ile - eoll < 8}.
By Example 19.7, the bracketing numbers of the classes of functions Mo satisfy
N[](E' 8 1I m llp,2' M 8 , L 2 (P») < C( r,
0<£ < 8.
The constant C is independent of £ and 8. The function Mo == 8m is an envelope function of
Mo. The left side also gives the bracketing numbers of the rescaled classes Mo /8 relative
to the envelope functions Mo/8 == m. Thus, we compute
(On
1[](8n, M8/ 8 , L 2 (P»):S Ja
dLog ( ) + LogC dE'.
The right side converges to zero as On i O uniformly in O. The envelope functions Mo / o == m
also satisfy the Lindeberg condition. The lernma follows from Theorem 19.28. .
19.6 MaximalInequalities
The main aim of this section is to derive the maximal inequality that is used in the proofs
of Theorems 19.5 and 19.28. We use the notation ;S for "srnaIler than up to a universal
constant" and denote the function 1 v log x by Log x.
19.6 Maximal Inequalities
285
A maximal inequality bounds the tail probabilities or moments of a supremum of random
variables. A maximal inequality for an infinite supremum can be obtained by combining
two devices: a chaining argument and maximal inequalities for finite maxima. The chaining
argument bounds every element in the supremum by a (telescoping) sum of small deviations.
In order that a sum of small terms is small, each of the terms must be exponentially small.
So we start with an exponential inequality. Next we apply this to obtain bounds on finite
suprema, and finally we derive the desired maximal inequality.
19.32 Lemma (Bernstein's inequality). For any bounded, measurable function ft
( 1 x 2 )
P ce; > x < 2 ex - -
p(1 nJ I ) - P 4 Pj2 + xIlJlloo/J1l '
every x > O.
ProoJ. The leading term 2 resuits from separate bounds on the right and left tail probabil-
ities. It suffices to bound the right tail probabilities by the exponential, because the left tail
inequality follows from the right tail inequality applied to - f. By Markov's inequality, for
every A > O,
( 00 1 ( A ) k ) n
P(GnJ > x) < e-AxEeAGnf = e- Ax 1 + t; k! J1l P(f - pf)k ,
by Fubini's theorem and next developing the exponential function in its power series. The
term for k == 1 vanishes because P (f - p f) == O, so that a factor 1 I n can be moved outside
the sum. We apply this inequality with the choice
1 x l ( x J1l )
A=2 Pj2+xll!lloo/J1l< 2 Pj2 1\ IIJlloo =:.1..11\.1..2,
Next, with Al and A2 defined as in the preceding display, we insert the bound A k < AIA -2 A
and use the inequality Ip(! - Pf)kl < Pf2(21IfII00)k-2, and we obtain
( ) n
1 00 1 1
P(Gnf > x) < e- AX 1 + - L ,-AX .
n k=2 k. 2
Because L(11 k!) < e - 2 < 1 and (1 + a)n < e an , the right side of this inequality is
bounded by exp( - AX 12), which is the exponential in the lemma. .
19.33 Lemma. For any finite class F of bounded, measurable, square-integrable func-
tions, with IFI elements,
II !II 00 J
EpIlGnll.F;Smax J1l log(l+IFI)+ max IIJIIP.2 log(l+IFI).
f n f
ProoJ. Define a == 2411!llooIJ1l and b == 24Pf2. For X > bla and x < bla the
exponent in Bernstein's inequality is bounded above by -3xla and -3x 2 Ib, respectively.
t The constant 1/4 can be replaced by 1/2 (which is the best possible constant) by amore precise argument.
286
Empirical Processes
For the truncated variables A f == G n f1 {IGnfl > bla} and B f == G n f1 {IGnfl < b la},
Bernstein's inequality yields the bounds, for all x > O,
( -3X )
P(IA f I > x) < 2 exp -----;;- ,
( 3X2 )
P(IBfl > x) < 2exp - b .
Combining the first inequality with Fubini's theorem, we obtain, with '1/1 p (x) == exp x P - 1,
( lA I ) [IAJlla [00
E1frl --!- = E lo eX dx = lo P(IAfl > xa) eX dx < 1.
By a similar argument we find that E '1/12 (I B f I I -JE) < 1. Because the function '1/11 is convex
and nonnegative, we next obtain, by Jensen's inequality,
1frl(Em.r IA;I ) < E1frl( max: I A f l ) < E 1frl C;I ) < IFI.
Because '1/1 1 1 (u) == log(l + u) is increasing, we can apply it across the display, and find a
bound on E max lA f I that yields the first term on the right side of the lemma. An analogous
inequality is valid for max f I B f I I -JE, but with 0/2 instead of '1/11. An application of the
triangle inequality concludes the proof. .
19.34 Lemma. For any class :F of m easurable functions f : X 1---+ such that P f2 < 8 2
Jar every J, we have, with a(8) = 8/ J Log N[J (8, F, L 2 (P)), and F an envelope function,
E IIG n IIF J[] (8, :F, L 2 (P)) + -JflP* F {F > -Jfla(8)}.
Proof. Because IGnfl < -Jfl(IPn + P)g for every pair of functions Ifl < g, we obtain,
for F an envelope function of :F,
E*IIGnJ{F> -Jfl a (8)}IIF < 2-JflPF{F > -Jfla(8)}.
The right side is twice the second term in the bound of the lemma. It suffices to bound
E* II Gnf {F < -Jfla(8)} IIF by a multiple of the first term. The bracketing numbers of the
class of functions f {F < a (8),Jn} if f ranges over :F are smaller than the bracketing
numbers of the class:F. Thus, to simplify the notation, we can assume that every f E :F is
bounded by ,Jna(8).
Fix an integer qo such that 48 < 2- qo < 88. There exists a nested sequence of partitions
:F == U 1 :F qi of :F, indexed by the integers q > qo, into N q disjoint subsets and measurable
functions qi < 2F such that
LTq .jLogN q {8 J LogN[](8,F,L 2 (P))d8,
q
sup If - gl < qi, P i < 2- 2q .
f,gEF qi
To see this, first cover :F with minimal numbers of L 2 (P)-brackets of size 2- q and re-
place these by as many disjoint sets, each of them equa1 to a bracket minus "previous"
brackets. This gives partitions that satisfy the conditions with qi equal to the difference
19.6 Maximal Inequalities
287
of the upper and lower brackets. If this sequence of partitions does not yet consist of suc-
cessive refinements, then replace the partition at stage q by the set of all intersections of
the form n =qo:Fp,ip' This gives partitions into N q == N qO . . . N q sets. Using the inequal-
ity (log TI Np) 1/2 < z= (log Np) 1/2 and rearranging sums, we see that the first of the two
displayed conditions is still satisfied.
Choose for each q > qo a fixed element fqi from each partitioning set :F qi , and set
Jr q f == fq i ,
q f == q i , if f E :Fq i .
Then Jr q f and q f run through a set of N q functions if f runs through:F. Define for each
fixed n and q > qo numbers and indicator functions
a q == 2- q / J Log N q + 1 ,
Aq-1f == 1{ qof < -J]ia qO ' ..., q-1f < -J]ia q -1},
Bq f == 1 { qO f < -J]ia qO ' . . . , q-1 f < -J]ia q -1, q f > -J]ia q }.
Then Aq f and Bq f are constant in f on each of the partitioning sets :F qi at level q, because
the partitions are nested. Our construction of partitions and choice of qo also ensure that
2a(8) < a qO ' whence Aqoi == 1. Now decompose, pointwise in x (which is suppressed in
the notation),
00 00
f - Jrqof == L(f - Jrqf)Bqf + L(17: q i - 17: q -1f)Aq-1f.
qo+1 qo+1
The idea here is to write the left side as the sum of f - 17: ql f and the telescopic sum
Z= +1 (17: q f - 17: q -1 f) for the largest q1 == q1 (f, x) such that each of the bounds q f on
the "links" 17: q i - 17: q -l f in the "chain" is uniformly bounded by -J]ia q (with ql possibly
infinite). We note that either all Bq f are 1 or there is a unique q1 > qo with B q1 I == 1. In
the first case Aq i == 1 for every q; in the second case Aq i == 1 for q < ql and Aq f == O
for q > q1.
N ext we apply the empirical process G- n to both series on the right separately, take
absolute values, and next take suprema over f E :F. We shall bound the means of the
resulting two variables.
First, because the partitions are nested, q f Bq f < q-1 f Bq f < -J]ia q -l trivially
p( qi)2Bqf < 2- 2q . Because lG-nil < G-ng + 2-J]iPg for every pair of functions
Iii < g, we obtain, by the triangle inequality and next Lemma 19.33,
00
E* L G-n(f - 17: q f) B q l
qo+l
00 00
< LE*IIG-n qfBqfIlF+ L2-J]iIlP qfBqfIIF
F qo+l qo+l
f: [ a q - 1 Log N q + r q .j Log N q + : r 2q ].
qo+ 1 q
In view of the defi nition o f a q , the series on the right can be bounded by a multiple of the
series Z=:+ 1 2 -q J Log N q .
288
Empirical Processes
Second, there are at most N q functions n q f - n q -1 f and at most N q -1 indicator functions
Aq-1f. Because the partitions are nested, the function Inqf - nq-1fIAq-1f is bounded
by £lq-1f Aq-1f < a q -1' The L 2 (P)-norm of Inqf - nq-lfl is bounded by 2- q + 1 .
Apply Lemma 19.33 to find
00
E* L Gn(nqf - n q -1f)A q - 1 f
qo+1
00
;S L [a q -1 Log N q + 2- q J Log N q ].
F qo+1
Again this is bounded above by a multiple of the series L:+1 2- q J Log N q .
To conclude the proof it suffices to consider the term s nqO f. Because Inqo f I < F <
a(8) < aqO and P (n qO f)2 < 8 2 by assumption, another application of Lemrna 19.33
yields
E* IIGnn qo f IIF ;S aqOLog N qO + 8 J Log N qo .
By the choi ce of qo , this is bounded by a multiple of the first few terms of the series
L:+1 2 - q J LogN q . .
19.35 Corollary. For any class :F of measurable functions with envelope function F,
E IIG n IIF ;S J[] (II F II P,2, :F, L 2 (P)).
Proof. Because:F is containedin the single bracket [- F, F], wehave N[] (8, :F, L 2 (P)) :=:
1 for 8 :=: 2I1Fllp,2' Then the constant a(8) as defined in the preceding lemmareduces to a
multiple of IIFllp,2, and P* F {F > a(8)} is bounded above by amultiple of IIFllp,2,
by Markov' s inequality. .
The second term in the maximal inequality Lemma 19.34 results from a crude majoriza-
tion in the first step of its pro of. This bound can be improved by taking special properties of
the class of functions :F into account, or by using different norms to measure the brackets.
The following lemmas, which are used in Chapter 25, exemplify this. t The first uses the
L 2 (P)-norm but is limited to uniformly bounded classes; the second uses a stronger norm,
which we call the "Bernstein norm" as it relates to a strengthening of B ernstein' s inequality.
Actually, this is not a true norm, but it can be used in the same way to measure the size of
brackets. It is defined by
Ilfll ,B :=: 2P(e 1fl - 1 - Ifl).
19.36 Lemma. For any class :F of measurable functions f : X 1--+ IR such that P f2 < 8 2
and Ilflloo < M for every f,
( J[](8,:F,L2(P)) )
E IIGnIlF ::J[](8,F,L2(P)) 1+ 82 M.
t For a proof of the following lemmas and further resuits, see Lemmas 3.4.2 and 3.4.3 and Chapter 2.14, in [146]
Also see [14], [15], and [51].
Problems
289
19.37 Lemma. For any class :F of measurable functions f : X 1---+ JR such that II f /I P,B
< 8 for every f,
E* II G II < J ( 8 :F II . II ) ( 1 + J[] ( 8 , :F, II . II P,B) )
p n F rv [] "P,B 8 2 .
Instead of brackets, we may also use uniform covering numbers to obtain maximal
inequalities. As is the case for the Glivenko-Cantelli and Donsker theorem, the inequality
given by Corollary 19.35 has a complete uniform entropy counterpart. This appears to be
untrue for the inequality given by Lemma 19.34, for it appears difficult to use the information
that a class :F is contained in a small L2(P)-ball directly in a uniform entropy maximal
inequality. t
19.38 Lemma. For any suitably measurable class :F of measurable functions f : X 1---+ JR,
we have, with e; == sup fEF IP> n f2 jIP> n F2 ,
E IIG n 11.1';S E( J (en, :F, L 2 ) /I F IIIP>n,2) ;S J (1, :F, L 2 ) II F II P,2.
Notes
The law of large numbers for the empirical distribution function was derived by Glivenko
[59] and Cantelli [19] in the 1930s. The Kolmogorov-Smimov and Cramer-von Mises
statistics were introduced and studied in the same period. The limit distributions of these
statistics were obtained by direct methods. That these were the same as the distribution
of corresponding functions of the Brownian bridge was noted and proved by Doob before
Donsker [38] formalized the the ory of weak convergence in the space of continuous func-
tions in 1952. Donsker's main examples were the empirical process on the real line, and
the partial sum process. Abstract empirical processes were studied more recently. The
bracketing central limit presented here was obtained by Ossiander [111] and the uniform
entropy central limit theorem by Pollard [116] and Kolčinskii [88]. In both cases these
were generalizations of earlier results by Dudley, who also was influential in developing
a theory of weak convergence that can deal with the measurability problems, which were
partly ignored by Donsker. The maximal inequality Lemma 19.34 was proved in [119].
The first Vapnik -Červonenkis classes were considered in [147].
For further results on the classical empirical process, including an introduction to strong
approximations, see [134] . For the abstract empirical process, see [57], [117], [120] and
[146]. For connections with limit theorems for random elements with values in Banach
spaces, see [98].
PROBLEMS
1. Derive a formula for the covariance function of the Gaussian process that appears in the limit of
the modified Kolmogorov-Smimov statistic for estimating normality.
t For a proof of the following lemma, see, for example, [120], or Theorem 2.14.1 in [146].
290
Empirical Processes
2. Find the covariance function of the Brownian motion process.
3. If Z is a standard Brownian motion, then Zet) - tZ(l) is a Brownian bridge.
4. Suppose that XI, . . . , X m and YI, . . . , Y n are independent samples from distribution functions F
and G, respectively. The Kolmogorov-Smirnov statistic for testing the null hypothesis Ha : F ==
G is the supremum distance Km,n == IIJF m - CG n 1100 between the empirical distribution functions
of the two samples.
(i) Find the limit distribution of Km,n under the null hypothesis.
(ii) Show that the Kolmogorov-Smirnov test is asymptotically consistent against every alterna-
tive F =f. G.
(iii) Find the asymptotic power function as a function of (g, h) for alternative s (F gj vm' G hj ft.)
belonging to smooth parametric models () Fe and () Ge.
5. Consider the class of all functions f : [O, 1] [O, 1] such that I f (x) - f (y) I < Ix - y I. Construct
a set of s-brackets for this class offunctions of cardinality bounded by exp(C/s).
6. Determine the VC index of
(i) The collection of all cells (a, b] in the real line;
(ii) The collection of all cells (-00, t] in the plane;
(iii) The collection of all translates {1/1(' - ()) : () E JR} of amonotone function 1/1 : JR JR.
7. Suppose that the class of functions F is VC. Show that the following classes are VC as well:
(i) The collection of sets {f > O} as f ranges over F;
(ii) The collection of functions x f (x) + g (x) as f ranges over F and g is fixed;
(iii) The collection of functions x f (x) g (x) as f ranges over F and g is fixed.
8. Show that a collection of sets is a VC class of sets if and only if the corresponding class of
indicator functions is a VC class of functions.
9. Let Fn and F be distribution functions on the real line. Show that:
(i) If Fn (x) F (x) for every x and F is continuous, then II Fn - F II 00 O.
(ii) If Fn(x) F(x) and Fn{x} F{x} for every x, then IIFn - Flloo O.
10. Find the asymptotic distribution of the mean absolute deviation from the median.
20
Functional Delta Method
The delta method was introduced in Chapter 3 as an easy way to turn
the weak convergence of a sequence of random vectors r n (Tn - e) into
the weak convergence oftransformations of the type rn(ep(Tn) - ep(e)).
It is useful to apply a similar technique in combination with the more
powerful convergence of stochastic processes. In this chapter we consider
the delta method at two levels. The first section is of a heuristic character
and limited to the case that Tn is the empirical distribution. The second
section establishes the delta method rigorously and in general, completely
parallel to the delta method for JRk, for Hadamard differentiable maps
between normed spaces.
20.1 von Mises Calculus
Let ]p n be the empirical distribution of arandom sample XI, . . . , X n from a distribution P.
Many statistics can be written in the fonn ep (:rP' n), where ep is a function that maps every
distribution of interest into some space, which for simplicity is taken equal to the real line.
Because the observations can be regained from :rP'n completely (unless there are ties), any
statistic can be expressed in the empirical distribution. The special structure assumed here
is that the statistic can be written as a fixed function ep of:rP' n, independent of n, a strong
assumption.
Because :rP' n converges to P as n tends to infinity, we may hope to find the asymptotic
behavior of ep (:rP' n) - ep (P) through a differential analysis of ep in a neighborhood of P. A
first -order analysis would have the fonn
<p (:rP' n) - <p (P) == ep (:rP' n - P) + . .. ,
where ep is a "derivative" and the remainder is hopefully negligible. The simplest approach
towards defining a derivative is to consider the function t <p (P + t H) for a fixed
perturbation H and as a function of the real-valued argument t. If ep takes its values in JR,
then this function is just a function from the reals to the reals. Assume that the ordinary
derivatives of the map t <p (P + t H) at t == O exist for k == 1, 2, . . . , m. Denoting them
by ep ) (H), we obtain, by Taylor's theorem,
1
ep (P + t H) - <p (P) == t<p (H) + . . . + -tm<p m) (H) + o(t m ).
m!
291
292
Functional Delta Method
Substituting t == 1/,Jn and H == G n , for G n == ,Jn (P n - P) the empirical process of the
observations, we obtain the von Mises expansion
1 I 1 1 Cm)
<p(Pn)-<P(P)== r;; <pp(G n )+...+, m/2 <Pp (G n )+....
'\In m.n
Actually, because the empirical process G n is dependent on n, it is not alegal choice for
Hunder the assumed type of differentiability: There is no guarantee that the remainder is
small. However, we make this our working hypothesis. This is reasonable, because the
remainder has one factor 1/,Jn more, and the empirical process G n shares at least one
property with a fixed H: It is "bounded." Then the asymptotic distribution of ep (P n) - ep (P)
should be determined by the first nonzero term in the expansion, which is usually the first-
order term ep (G n ). A method to make our wishful thinking rigorous is discussed in the next
section. Even in cases in which it is hard to make the differentation operation rigorous, the
von Mises expansion still has heuristic value. It may suggest the type of limiting behavior
of ep (P n) - ep (P), which can next be further investigated by ad-hoc methods.
We discuss this in more detail for the case that m == 1. A first derivative typically gives
a linear approximation to the original function. If, indeed, the map H ep (H) is linear,
then, writing P n as the linear combination P n == n-I L 8 Xi of the Dirac measures at the
observations, we obtain
1, 1 '
4J (W'n) - 4J (P) >:::; ,Jn 4J p (Crn) = n 6. 4J p (hi - P).
(20.1 )
Thus, the difference ep (P n) - ep (P) behaves as an average of the independent random
variables ep (8Xi - P). Ifthese variables have zero means and finite second moments, then
a normallimit distribution of ,Jn(ep(Pn) - ep(P)) may be expected. Here the zero mean
ought to be automatic, because we may expect that
1 4J (8x - P) dP(x) = 4J (I (8x - P) dP(X)) = 4J (O) = o.
The interchange of order of integration and application of ep is motivated by linearity (and
continuity) of this derivative operator.
The function x ep (8x - P) is known as the influence function of the function ep. It
can be computed as the ordinary derivative
d
ep (8x - P) == - ep( (1 - t)P + t8 x ).
dt It=O
The name "influence function" originated in developing robust statistics. The function
measures the change in the value ep (P) if an infinitesimally small part of P is replaced by a
pointmass at x. In robust statisties, functions and estimators with an unbounded influence
function are suspect, because a small fraction of the observations would have too much
influence on the estimator if their values were equal to an x where the influence function is
large.
In many examples the derivative takes the form of an "expectation operator" ep (H) ==
J $ p d H, for some function $ p with J $ p d P == O, at least for a subset of H. Then the
influence function is precisely the function $ p.
20.1 von Mises Calculus
293
20.2 Example (Mean). The sample mean is obtained as </J (IfD n) from the mean function
</J(?) == J s dP(s). The influence function is
<p (ox - P) = I S d[ (1 - t)P + t8 x ](s) == x - I S dP(s).
dt It=O
In this case, the approximation (20.1) is an identity, because the function is linear already. If
the sample space is a Euclidean space, then the influence function is unbounded and hence
the sample mean is not robust. O
20.3 Example (Wilcoxon). Let (X 1, YI), . . . , (X n, Y n ) be arandom sample from a bivari-
ate distribution. Write IF n and G n for the empirical distribution functions of the Xi and Y j ,
respectively, and consider the Mann- Whitney statistic
l In n
Tn == IFndGn == 2: LL 1 {X i < Yj}.
n i=1 j=1
This statistic corresponds to the function </J(F, G) == f F dG, which can be viewed as
a function of two distribution functions, or also as a function of a bivariate distribution
function with marginals F and G. (We have assumed that the sample sizes of the two
samples are equal, to fit the example into the previous discussion, which, for simplicity, is
restricted to i.i.d. observations.) The influence function is
<P(F,G)(OX,y - P) = 1 [(1 - t)F + t8x] d[ (1 - t)G + t8 y ]
d t It=O
= F(y) + 1 - G_(x) - 21 F dG.
The last step follows on multiplying out the two terms between square brackets: The function
that is to be differentiated is simply a parabola in t. For this case (20.1) reads
I IFn dlG n - I F dG t (F(Y i ) + 1 - G_(X i ) - 2 I F dG ) .
From the two-sample U -statistic theorem, Theorem 12.6, it is known that the difference
between the two sides of the approximation sign is actually o p (1/ ,Jn). Thus, the heuristic
calculus leads to the correct answer. In the next section an alternative proof of the asymptotic
normality of the Mann- Whitney statistic is obtained by making this heuristic approach
rigorous. D
20.4 Example (Z-functions). For every e in an open subset of }Rk, let x r+ O/() (x) be
a given, measurable map into JRk. The corresponding Z - function assigns to a probability
measure p a zero </J (P) of the map e r+ P o/(). (Consider only P for which a unique zero
exists.) If applied to the empirical distribution, this yields a Z -estimator </J (IfD n).
Differentiating with respect to t across the identity
o == (P + t8 x )o/q;(P+t8 x ) == Po/q;(P+t8 x ) + to/q;(P+t8 x ) (x),
and assuming that the derivatives exist and that e r+ o/() is continuous, we find
o == ( P1/1e ) [ <p(P + fOx) ] + o/q;(P) (x).
ae ()=q;(P) dt t=O
294
Functional Delta Method
The expression enclosed by squared brackets is the influence function of the Z - function.
Informally, this is seen to be equal to
( a ) -1
- -PljJ() ljJep(P) (X).
at) ()=ep(P)
In robust statistics we look for estimators with bounded influence functions. Because the
influence function is, up to a constant, equal to ljJ ep (P) (x), this is easy to achieve with
Z -estimators !
The Z -estimators are discussed at length in Chapter 5. The theorems discussed there
give sufficient conditions for the asymptotic normality, and an asymptotic expansion for
-Jn( ef> (P n) - ef> (P)). This is of the type (20.1) with the influence function as in the preceding
display. D
20.5 Example (Quantiles). The pth quantile of a distribution function Fis, roughly, the
number ef>(F) == F- 1 (p) such that FF- 1 (p) == p. We set Ft == (1 - t)F + t8x, and
differentiate with respect to t the identity
p == Ft Ft- 1 (p) == (1 - t) F ( Ft- 1 (p)) + t 8 x ( Ft- 1 (p) ) .
This "identity" may actually be only an inequality for certain values of p, t, and x, but we
do not worry about this. We find that
O = -F(F-1(p)) + f(F-1(p)) [ Pt-l(p) ] + 8 x (F-1(p)).
dt It=O
The derivative within square brackets is the influence function of the quantile function and
can be solved from the equation as
ep/ (8 - P) = _ 1[x,oo)(p- 1 (p)) - p .
F X f ( F -1 (p) )
The graph of this function is given in Figure 20.1 and has the following interpretation.
Suppose the pth quantile has been computed for a large sample, but an additional observation
x is obtained. If x is to the left of the pth quantile, then the pth quantile decreases; if x
is to the right, then the quantile increases. In both cases the rate of change is constant,
irrespective of the location of x. Addition of an observation x at the pth quantile has an
unstable effect.
P
f(F- 1 (p))
-1
F (p)
1-p
f(F- 1 (p))
Figure 20.1. Influence function of the pth quantile.
20.1 von Mises Calculus
295
The von Mises calculus suggests that the sequence of empirical quantiles ,Jfi (IF 1 (t) -
F- 1 (t)) is asymptotically normal with variance var F ep (o XI) == p (1 - p) / f o F- 1 (p)2.
In Chapter 21 this is proved rigorously by the delta method of the following section. Alter-
natively, a pth quantile may be viewed as an M -estimator, and we can apply the resuIts of
Chapter 5. O
20.1.1 Higher-Order Expansions
In most examples the analysis of the first derivative suffices. This statement is roughly
equivalent to the statement that most limiting distributions are norma!. However, in some
important examples the quadratic term dominates the von Mises expansion.
The second derivative ep (H) ought to correspond to a bilinear map. Thus, it is better
to write it as ep (H, H). If the first derivative in the von Mises expansion vanishes, then
we expect that
1 1 II 1 1 II
ep (P n) - ep (P) - -ep p (Crn, Crn) == - 2" L.....; L.....; ep p (o Xi - P, O X j - P).
2 n 2 n i=l j=l
The right side is a V-statistic of degree 2 with kern el function equal to hp (x, y) == 4ep (ox -
P,Oy - P). The kernel ought to be symmetric and degenerate in that Php(X, y) == O for
every y, because, by linearity and continuity,
1 4J;(8 x - P, 8y - P) dP(x) = 4J; (I (8x - P) dP(x), 8y - P )
== ep; (O, Oy - P) == O.
If we delete the diagonal, then a V -statistic turns into a U -statistic and hence we can apply
Theorem 12.10 to find the limit distribution of n (ep (P n) - ep(P)). We expect that
2 1 n
n ( ep (P n) - ep (P)) == - L L hp (Xi, X j) + - L hp (Xi, Xi) + O p (1) .
n .. n . 1
l<J 1=
If the function X 1-+ hp (x, x) is P - integrable, then the second term on the right only
contributes a constant to the limit distribution. Ifthe function (x, y) 1-+ h (x, y) is (P x P)-
integrable, then the first term on the right converges to an infinite linear combination of
independent X l- variables, according to Example 12.12.
20.6 Example (Cramer-von Mises). The Cramer-von Mises statistic is the function
ep (IF n) for ep (F) == f (F - FO)2 d Fo and a fixed cumulative distribution function Fo. By
direct calculation,
4J(F + tH) = 4J(F) + 2t 1 (F - Fa)H dFa + t 2 1 H 2 dFa.
Consequently, the first derivative vanishes at F == Fo and the second derivative is equal to
ep o (H) == 2 f H 2 d Fo. The von Mises calculus suggests the approximation
1 1 II 1 1 2
ep (IF n) - ep (Fo) - -ep]<; (CG n ) == - CG n d Fo.
2 n o n
296
Functional Delta Method
This is certainly correct, because it is just the detinition of the statistic. The preceding
discussion is still of some interest in that it suggests that the limit distribution is nonnormal
and can be obtained using the theory of V -statistics. Indeed, by squaring the sum that is
hidden in G , we see that
1 n n J
ncjJCIFn) == - LL (l xi ::: x - Fo(x)) ( lx r:s x - Fo(x))dFo(x).
n i=1 j=1
In Example 12.13 we used this representation to find that the sequence nep OF n) (1/6) +
L 1 j- 2 n- 2 (Z; - 1) for an i.i.d. sequence of standard normal variables ZI, Z2, . . ., if
the true distribution Fo is continuous. D
20.2 Hadamard-Differentiable Functions
Let Tn be a sequence of statistics with values in a normed space JI]) such that r n (Tn - ())
converges in distribution to alimit T, for a given, nonrandom (), and given numbers r n 00.
In the previous section the role of Tn was played by the empirical distribution JID n, which
might, for instance, be viewed as an element of the normed space D [ - 00, 00]. We wish
to prove that r n (ep (Tn) - cjJ(())) converges to a limit, for every appropriately differentiable
map CjJ, which we shall assume to take its values in another normed space IE.
There are several possibilities for detining differentiability of a map CjJ : II]) r+ IE between
normed spaces. A map ep is said to be Gateaux differentiable at e E II]) if for every fixed h
there exists an element CjJ (h) E IE such that
ep(() + th) - ep(()) == tep (h) + o(t),
as t {. O.
For JE the real line, this is precisely the differentiability as introduced in the preceding
section. Gateaux differentiability is also called "directional differentiability," because for
every possible direction h in the domain the derivative value ep (h) measures the direction
of the intinitesimal change in the value of the function cjJ. More formally, the o(t) term in
the previous displayed equation means that
ep(() + th) - ep(()) I
-ep()(h)
t
---+ O,
JE
as t {. O.
(20.7)
The suggestive notation CjJ (h) for the "tangent vectors" encourages one to think of the
directional derivative as a map ep : 10) 1-+ JE, which approximates the difference map ep (e +
h) - ep (()) : JI]) f---+ IE. It is usually included in the detinition of Gateaux differentiability that
this map CjJ : JI]) f---+ IE be linear and continuous.
However, Gateaux differentiability is too weak for the present purposes, and we need a
stronger concept. A map ep : II])cp f---+ IE, detined on a subset IlJ)cp of a normed space JI]) that
contains e, is called Hadamard differentiable at e if there exists a continuous, linear map
ep : JI]) 1-+ JE such that
4>(e +tht) -4>(e) -4>'(h)
t ()
---+ O,
lE
as t {. O, every ht h.
(More precisely, for every ht ---+ h such that e + tht is contained in the domain of CjJ
for all small t > O.) The values ep (h) of the derivative are the same for the two types
20.2 Hadamard-Differentiable Functions
297
of differentiability. The difference is that for Hadamard-differentiability the directions ht
are allowed to change with t Calthough they have to settle down eventually), whereas for
Gateaux differentiability they are fixed. The detinition as given requires that ep : Ir:» 1---+ JE
exists as a map on the whole ofIr:». Ifthis is not the case, but ep exists on a subset Ir:»o and the
sequences h t -+ h are restricted to converge to limits h E Ir:»o, then ep is called Hadamard
differentiable tangentially to this subset.
It can be shown that Hadamard differentiability is equivalent to the difference in C20.7)
tending to zero uniformly for h in compact subsets of JI]). For this reason, it is also called
compact differentiability. Because weak convergence of random elements in metric spaces
is intimately connected with compact sets, through Prohorov's theorem, Hadamard differ-
entiability is the right type of differentiability in connection with the delta method.
The derivative map C/J : JI]) 1---+ JE is assumed to be line ar and continuous. In the case of
tinite-dimensional spaces a line ar map can be represented by matrix multiplication and is
automatically continuous. In general, linearity does not imply continuity.
Continuity of the map C/J : JI]) 1---+ JE should not be confused with continuity of the depen-
dence e 1---+ ep Cif ep has derivatives in a neighborhood of e-values). If the latter continuity
holds, then ep is called continuously differentiable. This concept requires a norm on the set
of derivative maps but need not concern us here.
For completeness we discuss a third, stronger form of differentiability. The map ep :]])</> 1---+
JE is called Prechet differentiable at e if there exists a continuous, linear map ep : Ir:» 1---+ JE
such that
Ilepce + h) - ep ce) - ep Ch) IlE == o(llhll),
as II h II O.
Because sequences of the type t ht, as employed in the definition of Hadamard differentia-
bility, have norms satisfying Ilth t II == OCt), Frechet differentiability is the most restrictive
of the three concepts. In statistical applications, Frechet differentiability may not hold,
whereas Hadamard differentiability does. We did not have this problem in Section 3.1,
because Hadamard and Frechet differentiability are equivalent when ]]) == }Rk.
20.8 Theorem (Delta method). Let]]) and JE be normed linear spaces. Let ep : Ir:»</> c
JI]) 1---+ JE be Hadamard differentiable at e tangentially to JI])0. Let Tn : r2 n 1---+ ]])</> be maps
such that r n CTn - e) T for some sequence ofnumbers r n -+ 00 and arandom element T
that takes its values in JI])0. Then r n (ep CTn) - ep ce)) ep CT). If ep is defined and continuous
on the whole space ]]), then we also have r n (ep CTn) - ep ce)) == ep (rn CTn - e)) + O p Cl).
Proof. To prove that r n (ep CTn) -ep ce)) 'v'7 ep CT), de tine for each n amap gn Ch) == r n (ep ( e+
r;l h) - ep ce)) on the domain JI])n == {h : e + r;;l hE]])</>}. By Hadamard differentiability,
this sequence of maps satisfies gn' Ch n ,) -+ ep Ch) for every subsequence h n , -+ h E
]])0. Therefore, g n (r n CTn - e)) 'v'7 ep C T) by the extended continuous- mapping theorem,
Theorem 18.11, which is the first assertion.
The seemingly stronger last assertion of the theorem actually follows from this, if applied
to the function 1/1 == C ep, C/J ) : ]]) 1---+ JE x JE. This is Hadamard-differentiable at ce, e) with
derivative 1/1 == Cep , ep ). Thus, by the preceding paragraph, r n (1/1CT n ) -1j;ce)) converges
weakly to (ep CT), ep CT)) in JE x JE. By the continuous-mapping theorem, the diffe ence
rn(c/JCT n ) - epce)) - ep (rnCTn - e)) converges weakly to ep CT) - ep CT) == O. Weak
convergence to a constant is equivalent to convergence in probability. .
298
Functional Delta Method
Without the chain rule, Hadamard differentiability would not be as interesting. Con-
sider maps ep : llJ) f-* JE and 1j; : JE f-* JF that are Hadamard-differentiable at e and ep (e),
respectively. Then the composed map 1j; o ep : llJ) f-* ITf is Hadamard-differentiable at e, and
the derivative is the map obtained by composing the two derivative maps. (For Euclidean
spaces this means that the derivative can be found through matrix multiplication of the two
derivative matrices.) The attraction of the chain rule is that it allows a calculus of Hadamard-
differentiable maps, in which differentiability of a complicated map can be established by
decomposing this into a sequence of basic maps, of which Hadamard differentiability is
known or can be proven easily. This is analogous to the chain rule for real functions, which
allows, for instance, to see the differentiability of the map X f-* exp cos log( 1 + x2) in a
glance.
20.9 Theorem (Chain rule). Let <p : llJ)cp f-* JE1/! and 1j; : JE1/! f-* JF be maps defined on sub-
sets J.[J)cp and JE1/1 of normed spaces J.[J) and JE, respectively. Let ep be Hadamard-differentiable
at e tangentially to llJ)o and let 1j; be Hadamard-differentiable at ep (e) tangentially to ep (J.[J)o).
Then 1j; o ep : J.[J)cp f-* JF is Hadamard-differentiable at e tangentially to [J)o with derivative
1j; ((}) o ep .
Proof. Take an arbitrary converging path ht --+ h in J.[J). With the notation gt == t- 1 (ep (e +
tht) - ep (e)), we have
1j; o ep(e + thr) -1j; o ep(e) 1j; (ep (e) + tgt) -1/J(<tJ(e))
t t
By Hadamard differentiability of ep, gt --+ ep (h). Thus, by Hadamard differentiability of
1j;, the whole expression goes to 1j; ((}) (ep (h)). .
20.3 Some Examples
In this section we give examples of Hadamard-differentiable functions and applications of
the delta method. Further examples, such as quantiles and trimmed means, are discussed
in separate chapters.
The Mann- Whitney statistic can be obtained by substituting the empirical distribution
functions of two samples of observations into the function (F, G) f-* f F dG. This
function also plays a role in the construction of other estimators. The following lemma
shows that it is Hadamard-differentiable. The set B V M [a, b] is the set of all cadlag functions
z : [a, b] f-* [- M, M] C IR of variation bounded by M (the set of differences of ZI - Z2 of
two monotonely increasing functions that together increase no more than M).
20.10 Lemma. Let ep : [O, 1] f-* IR be twice continuously differentiable. Then the func-
tion (F I , F 2 ) f-* f ep (F I ) d F 2 is Hadamard-differentiable from the domain D[ -00, 00] X
B VI [-00, 00] C D[ -00,00] X D[ -00, 00] into IR at every pair offunctions of bounded
variation (F I , F 2 ). The derivative is given byt
(h), h 2 ) 1--+ h 2 1J o F)I'!:'oo - f h 2 - d1J o F) + f 1J/ (Fdh) dF 2 .
t We denote by h_ the left-continuous version of a eadlag funetion h and abbreviate hl = heb) - h(a).
20.3 Some Examples
299
Furthermore, thefunction (F I , F 2 ) 1---+ J(-oo,.] ep (F I ) dF 2 is Hamamard-differentiable as a
map into D[ -00,00].
Proof. Let hIt -+ hI and h 2t -+ h 2 in D[ -00,00] be such that F 2t == F 2 + th 2t is a
function of variation bounded by 1 for each t. Because F 2 is of bounded variation, it follows
that h 2t is of bounded variation for every t. Now, with FIt == F I + th lt ,
(I (jJ(Flt)dF 2t - 1 (jJ(F t )dF 2 )
1 ( <J;(Flt) - ep (F I ) , ) 1 1 '
== t - ep (FI)h I d F 2t + ep (F I ) dh 2t + ep (FI)h I d F 2t .
By partial integration, the second term on the right can be rewritten as ep o F l h 2t I oo -
J h 2t - d ep o F I . Under the assumption on h 2t , this converges to the first part of the derivative
as given in the lemma. The first term is bounded above by (II ep"ll rxJ II hIt 1100 + II ep' II 00 II h It -
hI 11(0) J dl F 2t I. Because the measures F 2t are of total variation at most 1 by assumption,
this expression converges to zero. To analyze the third term on the right, take a grid Uo ==
-00 < Ul < . . . < Um == 00 such that the function ep' o F I hI varies less than a prescribed
value £ > O on each interval [u i-I, U i)' Such a grid exists for every element of D [ - 00, 00]
(problem 18.6). Then
f (jJ'(Fdht d(F2t - F 2 ) < £(1 dIF2tl + d 1F 2 1 )
m+I
+ L I (ep' o F 1 hI) (u i-I) II F 2t [U i -1, U i) - F 2 [U i -1, U i) I.
i=I
The first term is bounded by £ O (1), in which the £ can be made arbitrarily small by the
choiee of the partition. For each fixed partition, the second term converges to zero as t O.
Renee the left side converges to zero as t O.
This proves the first assertion. The second assertion follows similarly. .
20.11 Example (Wilcoxon). Let JF m and G n be the empirical distribution functions of
two independent random samples XI, . . . , X m and YI, . . . , Y n from distribution functions
F and G, respeetively. As usual, consider both m and n as indexed by a parameter v, let
N == m + n, and assume that mj N -+ A E (O, 1) as v -+ 00. By Donsker's theorem and
Slutsky's lemma,
( GF GG )
vN(JF m - F, G n - G) r:; ' ,
v A ,J I - A
in the space D[ -00,00] X D[ -00,00], for a pair of independent Brownian bridges G F
and GG. The preceding lemma together with the delta method imply that
(I Fm dG n - 1 F dG ) - 1 J G A dF + 1 dG.
The random variable on the right is a continuous, linear function applied to Gaussian
processes. In analogy to the theorem that a linear transformation of a multivariate Gaussian
vector has a Gaussian distribution, it can be shown that a continuous, linear transformation
of a tight Gaussian process is normally distributed. That the present variable is normally
300
Functional Delta Method
distributed can be more easily seen by applying the delta method in its stronger form, which
implies that the limit variable is the limit in distribution of the sequence
- f .;N(G n - G)_ dF + f .;N(Yi m - F) dG.
This can be rewritten as the difference of two sums of independent random variables, and
next we can apply the central limit theorem for real variables. O
20.12 Example (Two-sample rank statisties). Let IHIN be the empirical distribution func-
tion of a sample XI, . . . , X m , YI, . . . , Y n obtained by "pooling" two independent random
samples from distributions F and G, respectively. Let R NI , . . . , RN N be the ranks of the
pooled sample and let CG n be the empirical distribution function of the second sample. If
no observations are tied, then NIHI N (Y j ) is the rank of Y j in the pooled sample. Thus,
N ( )
1 R N "
f 4J(IHI N ) dCGn == - L 4J
n" +1 N
J=m
is a two-sample rank statistic. This can be shown to be asyrnptotically normal by the
preceding lernma. Because NIHI N == mIF m + nCG n , the asymptotic normality of the pair
(IHI N , CG n ) can be obtained from the asymptotic normality of the pair (IF m, CG n ), which is
discussed in the preceding example. O
The cumulative hazard function corresponding to a cumulative distribution function F
on [O, 00] is defined as
1 dF
AF(t) == .
[O,t] 1 - F_
In particular, if F has a density f, then A F has a density AF == f/(l- F). If F(t) gives the
probability of "survivai" of a person or object until time t, then dAF(t) can be interpreted
as the probability of "instant death at time t given survival until t." The hazard function is
an important modeling tool in survival analysis.
The correspondence between distribution functions and hazard functions is one-ta-one.
The cumulative distribution function can be explicitly recovered from the cumulative hazard
function as the product integral of -A (see the proof of Lemma 25.74),
1- FA(t) == rI (1 - A{s})e-AC(t).
O<s::st
(20.13)
Here A {s} is the jump of A at s and AC (s) is the continuous part of A.
Under some restrictions the maps F A F are Hadamard differentiable. Thus, from
an asymptotic-statistical point of view, estimating a distribution function and estimating a
cumulative hazard function are the same problem.
20.14 Lemma. Let Dcp be the set of all nondecreasing cadlag functions F : [O, T] IR
with F (O) == O and 1 - F (T) > £ > O for sante £ > O, and let Elf be the set of all
nondecreasing cadlag functions A : [O, T] IR with A (O) == O and A (T) < M for some
M E IR.
20.3 Some Examples
301
(i) The map ep: llJ)<jJ c D[O, T] r-+ D[O, T] defined by ep(F) == A F is Hadamard differ-
entiable.
(ii) The map 1jJ :JEljf c D[O, T] r-+ D[O, T] defined by 1jJ(A) == F A is Hadamard differ-
entiable.
Proof. Part (i) follows from the chain rule and the Hadamard differentiability of each of
the three maps in the decomposition
F r-+ (F, 1 - F_) r-+ ( F, 1 ) r-+ 1 dF .
1 - F - [O,t] 1 - F_
The differentiability of the first two maps is easy to see. The differentiability of the last one
follows from Lemma 20.10. The proof of (ii) is longer; see, for example, [54] or [55]. .
20.15 Example (Nelson-Aalen estimator). Consider estimating a distribution function
based on right-censored data. We wish to estimate the distribution function F (or the corre-
sponding cumulative hazard function A) of arandom sample of "failure times" TI, . . . , Tn.
Unfortunately, instead of Ti we only observe the pair (Xi, i), in which Xi == Ti /\ C i is
the minimum of Ti and a "censoring time" C i , and i == 1 {Ii < C i } records whether Ii is
censored ( i == O) or not ( i == 1). The censoring time could be the closing date of the
study or a time that a patient is lost for further observation. The cumulative hazard function
of interest can be written
1 1 1 1
A(t) == dF == dH I ,
[O,t] 1 - F_ [O,t] 1 - H_
for 1 - H == (1 - F)(1 - G) and dH I == (1 - G_)dF, and every choice of distribution
function G. If we assume that the censoring times Cl, . . . , C n are arandom sample from
G and are independent of the failure times Ii, then H is precisely the distribution function
of Xi and HI is a "subdistribution function,"
1 - H(x) == P(X i > x),
HI(X) == P(X i < X, i == 1).
-
An estimator for A is obtained by estimating these functions by the empirical distributions
of the data, given by IHIn (x) == n-I "L7=I 1 {Xi < x} and IHI In (x) == n-I "L7=I 1 {Xi <
X, i == I}, and next substituting these estimators in the formula for A. This yields the
Nelson-Aalen estimator
A 1 1
An (t) == dIHI In .
[O,t] 1 - IHIn-
Because they are empirical distribution functions, the pair (IHIn, IHI In ) is asymptotically
normal in the space D [ - 00, 00] x D [ - 00, 00]. The easiest way to see this is to consider
them as continuous transformations of the (bivariate) empirical distribution function of the
pairs (Xi, i). The Nelson-Aalen estimator is constructed through the maps
(A, B) r-+ (1 - A, B) r-+ ( 1 , B ) r-+ 1 1 dB.
1 - A [O,t] 1 - A_
These are Hadamard differentiable on appropriate domains, the main restrictions being that
1 - A should be bounded away from zero and B of uniformly bounded variation. The
302
Functional Delta Method
asymptotic nonnality of the Nelson-Aalen estimator An (t) follows for every t such that
H(t) < l,andevenasaprocessinD[O,-r]foreveryi suchthatH(i) < 1.
If we apply the product integral given in (20.13) to the Nelson-Aalen estimator, then
we obtain an estimator 1 - F n for the distribution function, known as the product limit
estimator or Kaplan-Meier estimator. For a discrete hazard function the product integral is
an ordinary product over the jumps, by definition, and it can be seen that
A n # (J ': X. > x. ) - , n ( n - i ) (i)
1 - F n (t) == . ] - 1 1 == .
', X .< t #(J :X j > Xi) " X ' < t n -l + 1
1. 1 _ 1. (/)_
This estimator sequence is asymptotically normal by the Hadamard differentiability of the
product integral. D
Notes
A calculus of "differentiable statistical functions" was proposed by von Mises [104]. Von
Mises considered functions ep OF n) of the empirical distribution function (which he calls
the "repartition of the real quantities XI, . . . , X n ") as in the first section of this chapter.
Following Volterra he calls ep m times differentiable at F if the first m derivatives of the
map t 1-+ ep (F + t H) at t == O exist and have representations of the form
Ck) f f
epF (H) == ... 1/f(XI, ..., Xk) dH(XI)'.. dH(Xk).
This representation is motivated in analogy with the finite-dimensional case, in which H
would be a vector and the integrals sums. From the perspective of our section on Hadamard-
differentiable functions, the representation is somewhat arbitrary, because it is required that
a derivative be continuous, whence its general fonn depends on the norm that we use on
the domain of ep. Furthermore, the Volterra representation cannot be directly applied to, for
instance, alimiting Brownian bridge, which is not of bounded variation.
Von Mises' treatment is not at all informal, as is the first section of this chapter. After
developing moment bounds on the derivatives, he shows that n m / 2 (ep OF n) - ep (F) ) is asymp-
totically equivalent to ep m)(Gn) ifthe first m - 1 derivatives vanish at F and the (m + l)th
derivative is sufficiently regular. He refers to the approximating variables ep m) (CG n ), de-
generate V -statistics, as "quantics" and derives the asymptotic distribution of quantics of
degree 2, first for discrete observations and next in general by discrete approximation.
Hoeffding's work on U -statistics, which was published one year I ater, had a similar aim
of approximating complicated statistics by simpier ones but did not consider degenerate
U -statistics.
The systematic application of Hamadard differentiability in statistics appears to have
first been put forward in the (unpublished) thesis [125] of J Reeds and had a main focus
on robust functions. It was revived by Gill [53] with applications in survival analysis in
mind. With agrowing number of functional estimators available (beyond the empirical
distribution and product-limit estimator), the delta method is a simple but useful tool to
standardize asymptotic nonnality proofs.
Our treatment allows the domain llJ)cp of the map ep to be arbitrary. In particular, we do
not assume that it is open, as we did, for simplicity, when discussing the Delta method for
Problems
303
Euclidean spaces. This is convenient, because rnany functions of statistical interest, such
as zeros, inverses ar integrals, are defined only on irregularly shaped subsets of a norrned
space, which, besides a linear space, should be chosen big enough to support the limit
distribution of Tn.
PROBLEMS
1. Let ep(P) == f Jh(u, v) dP(u) dP(v) for a fixed given function h. The corresponding estimator
ep (JP n ) is known as a V -statistic. Find the influence function.
2. Findtheinfluencefunctionofthefunction<p(F) == f a(Fl +F2) dF2if Fl andF2 arethemarginals
of the bivariate distribution function F, and a is a fixed, smooth function. Write out ep (lF n). What
asymptotic variance do you expect?
3. Find the influence function of the map F 1--+ f[o,t] (1- F_)-l dF (the cumulative hazard function).
4. Show that a map ep : II]) 1--+ JE is Hadamard differentiable at a point e if and only if for every compact
set K C II]) the expression in (20.7) converges to zero uniformly in hEK as t O.
5. Show that the symmetrization map (e, F) 1--+ i ( F (t) + 1 - F (2e - t) ) is (tangentiall y) Hadamard
differentiable under appropriate conditions.
6. Let g : [a, b] 1--+ IR be a continuously differentiable function. Show that the map z 1--+ g o z with
domain the functions z : T 1--+ [a, b] contained in £00 (T) is Hadamard differentiable. What does
this imply for the function z 1--+ 1/ z?
7. Show that the map F 1--+ f[a,b] s dF(s) is Hadamard differentiable from the domain of all distri-
bution functions to IR, for each pair of finite numbers a and b. View the distribution functions as
a subset of D[ -00,00] equipped with supremum norm. What if a or b are infinite?
8. Find the first- and second-order derivative of the function ep(F) == f(F - FO)2 dF at F == Fo.
What limit distribution do you expect for <p (lF n) ?
21
Quantiles and Order Statistics
In this chapter we derive the asymptotic distribution of estimators of
quantiles from the asymptotic distribution of the corresponding estima-
tors of a distribution function. Empirical quantiles are an example, and
hence we also discuss some results concerning order statistics. Fur-
the rmo re, we discuss the asymptotics of the median absolute deviation,
which is the empirical1/2 -quantile of the observations centered at their
1/2 -quantile.
21.1 Weak Consistency
The quantile function of a cumulative distribution function F is the generalized inverse
F- 1 : (O, 1) JR given by
F -1 (p) == inf { x: F (x) > p}.
It is a left -continuous function with range equal to the support of F and hence is often
unbounded. The following lernma records some useful properties.
21.1 Lemma. For every O < P < 1 and x E JR,
(i) F- 1 (p) < x ifjp < F(x),o
(ii) F o F- 1 (p) > p with equality ifj p is in the range of F,o equality can fail only if
F is discontinuous at F- 1 (p);
(iii) F_ o F- 1 (p) < p;
(iv) F- 1 o F (x) < x,o equality fails ifj x is in the interior or at the right end of a "fiat"
of F,o
(v) F- 1 o F o F- 1 == F- 1 ,o F o F- 1 o F == F,o
(vi) (F o G)-l == G- 1 o F-].
Proof. The proofs of the inequalities in (i) through (iv) are best given by a picture. The
equalities (v) follow from (ii) and (iv) and the monotonicity of F and F- 1 . If P == F(x)
for some x, then, by (ii) p < F o F- 1 (p) == F o F- 1 o F(x) == F(x) == p, by (iv). This
proves the first statement in (ii) the second is immediate from the inequalities in (ii) and
(iii). Statement (vi) follows from (i) and the definition of (F o G)-l. .
304
21.2 Asymptotic Normality
305
Consequences of (ii) and (iv) are that F o F- I (p) - p on (O, 1) if and only if F is
continuous (i.e., has range [O, 1]), and F- I o F(x) - x on IR if and only if F is strictly
increasing (i.e., has no "flats"). Thus F- I is a proper inverse if and only if F is both
continuous and strictly increasing, as one would expect.
By (i) the random variable F- I (U) has distribution function F if U is uniformly dis-
tributed on [O, 1]. This is called the quantile transformation. On the other hand, by (i) and
(ii) the variable F (X) is uniformly distributed on [O, 1] if and only if X has a continuous
distribution function F. This is called the probability integral transformation.
A sequence of quantile functions is defined to converge weakly to alimit quantile func-
tion, denoted Fn- I F- I , if and only if Fn- I (t) ---* F- I (t) at every t where F- I is contin-
uous. This type of convergence is not only analogous in form to the weak convergence of
distribution functions, it is the same.
21.2 Lemma. For any sequence of cumulative distributionfunctions, Fn- I F- I if and
only if Fn F.
Proof. Let U be uniformly distributed on [O, 1]. Because F- I has at most countably many
discontinuity points, Fn- I F- I implies that Fn- I (U) ---* F- I (U) almost surely. Conse-
quently, Fn- I (U) converges in law to F- I (U), which is exactly Fn F by the quantile
transformation.
For a proof the converse, let V be a normally distributed random variable. If Fn F,
then Fn (V) F (V), because convergence can fail only at discontinuity points of F.
Thus <p (Fn- I (t) ) == P( Fn (V) < t) (by (i) of the preceding lemma) converges to P( F (V) <
t) == <P( F- I (t)) at every t at which the limit function is continuous. This includes
every t at which F- I is continuous. By the continuity of <p-I, Fn- I (t) ---* F- I (t) for every
such t. .
A statistical application of the preceding lemma is as follows. If a sequence of estimators
Fn of a distribution function F is weakly consistent, then the sequence of estimators Fn- I is
weakly consistent for the quantile function F- I .
21.2 Asymptotic Normality
In the absence of information conceming the underlying distribution function F of a sample,
the empirical distribution function 1F n and empirical quantile function 1F;;-1 are reasonable
estimators for F and F- I , respectively. The empirical quantile function is related to the
order statistics X n (1), . . . , Xn(n) of the sample through
1F;I (p) == Xn(i),
( i -1 i ]
for p E --;;-' n .
One method to prove the asymptotic normality of empirical quantiles is to view them as
M -estimators and apply the theorems given in Chapter 5. Another possibility is to express
the distribution function P(Xn(i) < x) into binomial probabilities and apply approximations
to these. The method that we follow in this chapter is to deduce the asymptotic normality of
quantiles from the asymptotic normality of the distribution function, using the delta method.
306
Quantiles and Order Statistics
An advantage of this method is that it is not restricted to empirical quantiles but applies to
the quantiles of any estimator of the distribution function.
For a nondecreasing function FED [a, b], [a, b] c [- 00, 00], and a fixed p E ffi., let
ep (F) E [a, b] be an arbitrary point in [a, b] such that
F(ep(F)-) < p < F(ep(F)).
The natural domain ID\p of the resulting map ep is the set of all nondecreasing F such that
there exists a solution to the pair of inequalities. If there exists more than one solution, then
the precise choice of ep (F) is irrelevant. In particular, ep (F) may be taken equal to the pth
quantile F- I (p).
21.3 Lemma. Let F E 10\1> be differentiable at a point P E (a, b) such that F( p) == p,
with positive derivative. Then ep :]D)</J c D[a, b] r-+ ffi. is Hadamard-differentiable at F
tangentially to the set of functions h E D [a, b] that are continuous at p, with derivative
ep (h) == -h( p)1 F!( p)'
Proof. Let ht ---+ h uniformly on [a, b] for a function h that is continuous at p' Write pt
for ep(F + th t ). By the definition of ep, for every St > O,
(F + tht)( pt - St) < P < (F + tht)( pt).
Choose St positive and such that St == o(t). Because the sequence ht converges uniformly
to a bounded function, it is uniformly bounded. Conclude that F( pt - St) + O(t) < p <
F( pt) + O(t). By assumption, the function F is monotone and bounded away from p
outside any interval ( p - S, p + s) around p' To satisfy the preceding inequalities the
numbers pt must be to the right of p - S eventually, and the numbers pt - St must be to
the left of p + S eventually. In other words, pt ---+ p'
By the uniform convergence of ht and the continuity of the limit, ht( pt - St) ---+ h( p)
for every St ---+ o. Using this and Taylor's formula on the preceding display yields
p + ( pt - p)F! ( p) - o( pt - p) + O(St) + th( p) - o(t)
< p < p + ( pt - p)F! ( p) + o( pt - p) + O(St) + th( p) + o(t).
Conclude first that pt - p == O(t). Next, use this to replace the o( pt - p) term s in the
display by o(t) term s and conclude that ( pt - p)lt ---+ -(hi F')( p)' .
Instead of asingle quantile we can consider the quantile function F r-+ ( F- I (p ) ) ,
Pl <P<P2
for fixed numbers O < Pl < P2 < 1. Because any quantile function is bounded on an interval
[Pl, P2] strictly contained in (O, 1), we may hope that a quantile estimator converges in
distribution in £00 (Pl, P2) for such an interval. The quantile function of a distribution
with compact support is bounded on the whole interval (O, 1), and then we may hope to
strengthen the result to weak convergence in £00(0, 1).
Given an interval [a, b] c ffi., let]D)1 be the set of all restrictions of distribution functions
on ffi. to [a, b], and let ]D)2 be the subset Of]D)1 of distribution functions of measures that give
mass 1 to (a, b].
21.2 Asymptotic Normality
307
21.4 Lemma.
(i) Let O < Pl < P2 < 1, and let F be continuously differentiable on the interval [a, b] ==
[ F- l (Pl) - E, F- l (P2) + E ] for some E > O, with strictly positive derivative f. Then
the inverse map G r-+ G- l as a map IIJ)1 C D[a, b] r-+ fOO[Pl, P2] is Hadamard
differentiable at F tangentially to C[a, b].
(ii) Let F have compact support [a, b] and be continuously differentiable on its support
with strictly positive derivative f. Then the inverse map G r-+ G -1 as a map IIJ)2 C
D[a, b] r-+ fOO(O, 1) is Hadamard differentiable at F tangentially to C[a, b].
In both cases the derivative is the map h r-+ -(hi f) oF-I.
Proof. It suffices to make the proof of the preceding lemma uniform in p. We use the
same notation.
(i). Because the function F has apositive density, it is strictly increasing on an interval
[ p , p;] that strictly contains [ Pl' P2]. Then on [p , p;] the quantile function F- l is the
ordinary inverse of F and is (uniformly) continuous and strictly increasing. Let ht -+ h
uniformly on [ p ' p;] for a continuous function h. By the proof of the preceding lemma,
Pit Pi and hence every pt for Pl < P < P2 is contained in [ p , p;] eventually. The
remainder of the proof is the same as the proof of the preceding lemma.
(ii). Let ht -+ h uniformly in D[a, b], where h is continuous and F + tht is contained
in IIJ)2 for all t. Abbreviate F- l (p) and (F + t ht) -1 (p) to p and pt, respectively. Because
F and F + t ht are concentrated on (a, b] by assumption, we have a < pt, p < b for all
O < P < 1. Thus the numbers E pt == t 2 /\ ( pt - a) are positive, whence, by definition,
(F + tht)( pt - Ept) < P < (F + tht)( pt).
By the smoothness of F we have F( p) == p and F( pt -Ept) == F( pt) + O (E pt), uniformly
in O < P < 1. It follows that
-th( pt) + o(t) < F( pt) - F( p) < -th( pt - Ept) + o(t).
The o(t) terms are uniform in O < P < 1. The far left side and the far right side are O(t);
the expression in the middle is bounded above and below by a constant times I pt - p I.
Conclude that I pt - p I == O(t), uniform1y in p. Next, the lemma follows by the uniform
differentiability of F. .
Thus, the asymptotic normality of an estimator of a distribution function (or another
nondecreasing function) automatically entails the asymptotic normality of the correspond-
ing quantile estimators. More precisely, to derive the asymptotic normality of even asingle
quantile estimator Fn- l (p), we ne ed to know that the estimators Fn are asymptotically nor-
mal as a process, in a neighborhood of F- l (p). The standardized empirica1 distribution
function is asymptotically norma1 as a process indexed by IR, and hence the empirica1
quantiles are asymptoticall y norma!.
21.5 Corollary. Fix O < p < 1. If F is differentiable at F- 1 (p) with positive derivative
f ( F -1 (p) ), then
C ( -1 -1 ) 1 l{X i :::: F- 1 (p)} - P
v n JF n (p) - F (p) == - r:; ( -1 ) + O p ( 1 ) .
v n i=l f F (p)
308
Quantiles and Order Statistics
Consequently, the sequence (IF 1 (p) - F- 1 (p)) is asyrnptotically normal with mean O
and variance p(1 - p) I f2 (F- 1 (p)). Furthermore, if F satisfies the conditions (i) or (ii)
of the preceding lemma, then (IF l - F- 1 ) converges in distribution in £oo[Pl, P2] or
£00(0,1), respectively, to the process GAlf(F-1(p)), where GA is a standard Brownian
bridge.
Proof. By Theorem 19.3, the empirical process Gn,F == (IFn - F) converges in distri-
bution in D[-oo, 00] to an F-Brownian bridge process CG F == GA o F. The sample paths
of the limit process are continuous at the points at which F is continuous. By Lemma 21.3,
the quantile function F 1---+ F- 1 (p) is Hadamard-differentiable tangentially to the range
of the limit process. By the functional delta method, the sequence (IF 1 (p) - F -1 (p) )
is asymptotically equivalent to the derivative of the quantile function evaluated at G n , F,
that is, to -Gn,F (F- 1 (p)) I f (F- 1 (p)). This is the first assertion. Next, the asymptotic
normality of the sequence (IF 1 (p) - F- 1 (p)) follows by the central limit theorem.
The convergence of the quantile process follows similarly, this time using Lemma
21.4. .
21.6 Example. The uniform distribution function has derivative 1 on its compact support.
Thus, the uniform empirical quantile process converges weakly in loo (O, 1). The limiting
process is a standard Brownian bridge.
The normal and Cauchy distribution functions have continuous derivatives that are
bounded away from zero on any compact interval. Thus, the normal and Cauchy empirical
quantile processes converge in £oo[Pl, P2], for every O < Pl < P2 < 1. D
The empirical quantile function at a point is equal to an order statistic of the sample. In
estimating a quantile, we could also use the order statistics directly, not necessarily in the
way that IF ] picks them. For the kn - th order statistic X n (kn) to be a consistent estimator
for F- 1 (p), we need minimally that knin ---+ p as n ---+ 00. For mean-zero asymptotic
normality, we also need that kn I n ---+ p faster than 1 I , which is necessary to ensure that
Xn(k n ) and IF l (p) are asymptotically equivalent. This still allows considerable freedom for
choosing kn.
21.7 Lemma. Let F be differentiable at F-1(p) with positive derivative and let knin ==
p + c/ + o(I/ ). Then
p c
y'fi (Xn(k n ) - JF;;-I (p)) f (p-I (p))'
Proof. First as sume that F is the uniform distribution function. Denote the observations
by Ui, rather than Xi. Define a function gn: £00(0,1) 1---+ IR by gn(Z) ==z(knln) - z(p).
Then gn (Zn) ---+ z (p) - z (p) == O, whenever Zn ---+ z for a function z that is continuous at p.
Because the uniform quantile process (G l- G- 1 ) converges in distribution in £00(0, 1),
the extended continuous-mapping theorem, Theorem 18.11, yields gn ( (G l - G- 1 )) ==
(Un(kn) - G l(p)) - (knln - p) -v--+O. This is the result in the uniforrn case.
A sample from a general distribution function F can be generated as F- 1 (U i ), by the
quantile transformation. Then ( X n (kn) - IF 1 (p)) is equal to
y'fi[ p-I (Un(k n )) - p-I (p)] - y'fi[ p-I (G;;-I (p)) - p-I (p) J.
21.2 Asymptotic Normality
309
Apply the delta method to the two terms to see that f (F- I (p)) times their difference is
asymptotically equivalent to (Un(kn) - p) - (G;;l(p) - p). .
21.8 Example (Confidence intervals for quantiles). If XI, . . . , X n is arandom sample
from a continuous distribution function F, then Ul == F(X I ),..., Un == F(X n ) are a
random sample from the uniform distribution, by the probability integral transformation.
This can be used to construct confidence interval s for quantiles that are distribution-free
over the class of continuous distribution functions. For any given natural numbers k and I,
the interval (Xn(k), Xn(l)] has coverage probability
Pp(Xn(k) < F-I(p) < Xn(l)) == P(Un(k) < P < Un(l))'
Because this is independent of F, it is possible to obtain an exact confidence interval for
every fixed n, by determining k and I to achieve the desired confidence level. (Here we have
some freedom in choosing k and I but can obtain only finitely many confidence levels.) For
large n, the values k and I can be chosen equal to
k,l _ Jp(1 p)
- - p =l: Za .
n n
To see this, note that, by the preceding lemma,
G;; I (p) I p (1 - p) ( 1 )
Un(k), Un(l) = :I:: ZaV n + o p .
Thus the e vent Un(k ) < P < Un(l) is asymptotically equivalent to the event IG;;I(p)-
p I < Za y' P (1 - p). Its probability converges to 1 - 2a.
An alternative is to use the asymptotic normality of the empirical quantiles JF;; I, but
this has the unattractive feature of having to estimate the density f (F- I (p) ), because this
appears in the denominator of the asymptotic variance. If using the distribution- free method,
we do not even have to assume that the density exists. O
The application of the Hadamard differentiability of the quantile transformation is not
limited to empirical quantiles. For instance, we can also immediately obtain the asymp-
totic normality of the quantiles of the product limit estimator, or any other estimator of a
distribution function in semiparametric models. On the other hand, the results on empirical
quantiles can be considerably strengthened by taking special properties of the empirical
distribution into account. We discuss a few extensions, mostly for curiosity value. t
Corollary 21.5 asserts that Rn (p) O, for, with p == F- I (p),
Rn(P) == f( p) (JF;;I(p) - F-I(p)) + 01(Wn( p) - F( p)).
The expression on the left is known as the standardized empirical difference process. "Stan-
dardized" refers to the leading factor f ( p) . That a sum is called a difference is curious
but stems from the fact that minus the second term is approximately equal to the first term.
The identity shows an interesting symmetry between the empirical distribution and quantile
t See [134, pp. 586-587] for further information.
310
Quantiles and Order Statistics
processes, particularly in the case that F is uniform, if f ( p) = 1 and p = p. The result
that Rn (p) O can be refined considerably. If F is twice-differentiable at p with positive
first derivative, then, by the Bahadur- Kiefer theorems,
n 1/4 [ 32 ] 1/4
lim sup 3/4 1 Rn (p) I == - p (1 - p) ,
n--+oo (1og10g n) 27
2 1 00 ( X ) ( Y )
n 1 / 4 Rn(P)'V't <p - 1 dy.
,J p(l - p) o V'Y ,J p(l - p)
a. s. ,
The right side in the last display is a distribution function as a function of the argument
x. Thus, the magnitude of the empirical difference process is Op(n- 1 / 4 ), with the rate
of its fluctuations being equal to n- 1 / 4 (1oglog n)3/4. Under some regularity conditions on
F, which are satisfied by, for instance, the uniform, the normal, the exponential, and the
logistic distribution, versions of the preceding results are also valid in supremum norm,
n 1/4 1
lim sup 1/2 1/4 II Rn 1100 == M ' a.s.,
n--+oo (log n) (2log10g n) v 2
n 1 / 4
(log n) 1/2 II Rn II 00 "" J II ZA II 00'
Here ZA is a standard Brownian motion indexed by the interval [O, 1].
21.3 Median Absolute Deviation
The median absolute deviation of a sample XI, . . . , X n is the robust estimator of scale
defined by
MADn == med Xi - med Xi .
l::si::sn l::Si ::Sn
It is the median of the deviations of the observations from their median and is often recom-
mended for reducing the observations to a standard scale as a first step in a robust procedure.
Because the median is a quantile, we can prove the asymptotic normality of the median
absolute deviation by the delta method for quantiles, applied twice.
If a variable X has distribution function F, then the variable IX - () I has the distribution
function x r--+ F (() + x) - F _ (() - x). Let ((), F) 1--* 12 ((), F) be the map that assigns to
a given number () and a given distribution function F the distribution function F (() + x) -
F _ (() - x), and consider the function 1 == 13 o 12 o 11 defined by
F ((): == F- 1 (1/2), F) G: == F((} + .) - F_((} - .) G- 1 (1/2).
If we identify the median with the 1/2-quantile, then the median absolute deviation is
exactly 1 (lF n)' Its asymptotic normality follows by the delta method under a regularity
condition on the underlying distribution.
21.9 Lemma. Let the numbers mp and mG satisfy F(mp) == == F(mp + mG) - F(mp -
mG)' Suppose that F is differentiable at m p with positive derivative and is continuously
differentiable on neighborhoods of m p + mG and m p - mG with positive derivative at
21. 3 Median Absolute Deviation
311
m F + mG and/or m F - mG. Then the map ep: D[ -00, 00] IR, with as domain the dis-
tribution functions, is Hadamard-differentiable at F, tangentially to the set of functions
that are continuous both at mF and on neighborhoods ofmF + mG and mF - mG. The
derivative ep (H) is given by
H(mF) f(mF + mG) - f(mF - mG)
f(mF) f(mF + mG) + f(mF - mG)
H(mF + mG) - H(mF - mG)
f(mF + mG) + f(mF - mG)
Proof. Define the maps epi as indicated previously.
By Lemma 21.3, the map epl: D[ -00, 00] IR x D[ -00,00] is Hadamard-differenti-
able at F tangentially to the set of functions H that are continuous at m F.
The map ep2: IR x D[-oo, 00] D[mG - 8, mG + 8] is Hadamard-differentiable at the
point (m F, F) tangentially to the set of points (g, H) such that H is continuous on the
interval s [mF ::!: nlG - 28, mF ::!: mG + 28], for sufficiently small 8 > O. This follows
because, if at -+ a and Ht -+ H uniformly,
(F + tHt)(mF + tat + x) - F(mF + x)
-+ af(mF + x) + H(mF + x),
t
uniformly in x mG, and because a similar statement is valid for the differences (F +
tHt)-(mF + tat - x) - F_(mF - x). The range of the derivative is contained in C[mG -
8, mG + e].
Finally, by Lemma 21.3, the map ep3 : D[mG - 8, mG + 8] IR is Hadamard-differenti-
able at G == ep2(mF, F), tangentially to the set of functions that are continuous at mG,
because G has apositive derivative at its median, by assumption.
The lemma follows by the chain rule, where we ascertain that the tangent spaces match
up properly. .
The F-Brownian bridge process G F has sample paths that are continuous everywhere
that F is continuous. Under the conditions of the lemma, they are continuous at the point
mF and in neighborhoods of the points mF + mG and mF - mG. Thus, in view of the
lemma and the delta method, the sequence vn (ep (lF n) - ep (F)) converges in distribution to
the variable ep (G F).
21.10 Example (Symmetric F). If F has a density that is symmetric about O, then its
median mF is O and the median absolute deviation mG is equal to F- 1 (3/4). Then
the first term in the definition of the derivative vanishes, and the derivative ep (G F) at
the F-Brownian bridge reduces to - (GA (3/4) - G A (1/4))/2f(F- 1 (3/4)) for a stan-
dard Brownian bridge GA' Then the asymptotic variance of vn(MAD n - mG) is equal
to (1/16)/f o F- 1 (3/4)2. D
21.11 Example (Normal distribution). If F is equal to the normal distribution with mean
zero and variance 0- 2 , then m F == O and mG == o- <1>-1 (3/4). We find an asymptotic variance
(0- 2 /16)cjJ 0<1>-1 (3/4)-2. As an estimator for the standard deviation 0-, we use the estimator
MAD n /<P- 1 (3/4), and as an estimator for 0- 2 the square of this. By the delta method,
the latter estimator has asymptotic variance equal to (1/4)0-4ep o <1>-1(3/4)-2<1>-1(3/4)-2,
which is approximately equal to 5.440- 4 . The relative efficiency, relative to the sample
variance, is approximately equal to 37%, and hence we should not use this estimator without
a good reason. D
312
Quantiles and Order Statistics
21.4 Extreme Values
The asymptotic behavior of order statistics Xn(k n ) such that k n / n O or I is, of course,
different from that of central-order statistics. Because Xn(k n ) < X n means that at most n - kn
of the Xi can be bigger than X n , it follows that, with Pn == P(X i > x n ),
Pr(Xn(k n ) < X n ) == P(bin(n, Pn) < n - kn).
Therefore, limit distributions of general-order statistics can be derived from approximations
to the binomial distribution. In this section we consider the most extreme cases, in which
kn == n - kfor a fixed number k, starting with the maximum Xn(n). We write F(t) ==
P(X i > t) for the survival distribution of the observations, arandom sample of size n
from F.
The distribution function of the maximum can be derived from the preceding display, ar
directly, and satisfies
n ( n F ( n) ) n
P(Xn(n) < X n ) == F(x n ) == I - n .
This representation readil y yields the following lemma.
21.12 Lemma. Forany sequence ofnumbers X n and any T E [0,00], we have P(Xn(n) <
X n ) e-r: ifand only ifnF(x n ) T.
In view of the lemma we can find "interesting limits" for the probabilities P(Xn(n) < X n )
only for sequences X n such that F(xn) == OCI/n). Depending on F this may mean that X n
is bounded ar converges to infinity.
Suppase that we wish to find constants an and b n > O such that b;; 1 (Xn(n) - an) con-
verges in distribution to a nontrivial limit. Then we must choose an and b n such that
F(a n + bnx) == O(I/n) for a nontrivial set of x. Depending on F such constants mayor
may not exist. It is a bit surprising that the set of possible limit distributions is extremely
small. t
21.13 Theorem (Extremal types). If b;; 1 (Xn(n) - an) G for a nondegenerate cumula-
tive distribution function G, then G be long s to the location-scale family of a distribution of
one of the following forms:
(i) e-e- x with support JR;
(ii) e-(1l x C¥) with support [O, (0) and a > O;
(iii) e-(-x)C¥ with support (-00, O] and a > O.
21.14 Example (Uniform). If the distribution has finite support [O, 1] with F(t) == (1 -
t)a, then n F (1 + n- I/a X) (-x)a for every X < O. In view of Lernma 21.12, the
sequence n Ila (Xn(n) - 1) converges in distribution to a limit of type (iii). The uniform
distribution is the special case with a == I, for which the limit distribution is the negative
of an exponential distribution. D
t For a proof of the following theorem, see [66] or Theorem 1.4.2 in [90].
21.4 Extreme Values
313
21.15 Example (Pareto). The survival distribution of the Pareto distribution satisfies
F (t) == (fL / t yx for t > fL. Thus n F (n Ila fLx) --* 1/ x a for every X > O. In view of Lemma
21.12, the sequence n-Ila Xn(n)/ fL converges in distribution to alimit oftype (ii). O
21.16 Example (Norma 1). For the normal distribution the calculations are similar, but
more delicate. We choose
J 1 log log n + log 4n
an == 2log n - - ,
2 ,j 2logn
b n == 1/ ) 2 log n.
Using Mill's ratio, which asserts that <P(t) f"V cjJ(t)/t as t --* 00, it is straightforward to
se e that n<P(a n + bnx) --* e- X for every X. In view of Lemma 21.12, the sequence
,j 2log n(Xn(n) - an) converges in distribution to a limit of type (i). O
The problem of convergence in distribution of suitably normalized maxima is solved in
general by the following theorem. Let -rF == sup{t: F(t) < I} be the right endpoint of F
(possibly (0).
21.17 Theorem. There exist constants an and b n such that the sequence b;; 1 (Xn(n) - an)
converges in distribution if and only if, as t --* -rF,
(i) There exists a strictly positivefunction g on JR such that F(t + g(t)x)/ F(t) --* e- X ,
for every x E JR;
(ii) 7:F == 00 and F(tx)/ F(t) --* l/x a ,for every x > O;
(iii) 7:F < 00 and F ( -rF - (-rF - t)x)/ F (t) --* x a , for every x > O.
The constants (an, b n ) can be taken equal to (un, g(u n )), (O, un) and (-rF, -rF - un), respec-
tively, for Un == F- I (1 - lin).
Proof. We only give the proof for the "only if" part, which follows the same lines as
the preceding examples. In every. of the three cases, nF(u n ) --* 1. To see this it suffices
to show that the jump F(u n ) - F(u n -) == o(l/n). In case (i) this follows because, for
ev ery x < O, the jump is smaller than F (u n + g(un)x) - F (u n ), which is of the order
F(un)(e- X - 1) < (l/n)(e- X - 1). The right side can be made smaller than E(l/n)
for any E > O, by choosing x close to O. In case (ii), we can bound the jump at Un by
- - -
F(xu n ) - F(u n ) for every x < 1, whichisoftheorder F(u n )(1/x a -1) < (1/n)(1/x a -1).
In case (iii) we argue similarly.
We conclude the proof by applying Lemma 21.12. For instance, in case (i) we have
n F (u n + g(un)x) f"V n F (un)e- X --* e- x for every x, and the result follows. The argument
under the assumptions (ii) or (iii) is similar. .
If the maximum converges in distribution, then the (k + l)-th largest-order statistics
Xn(n-k) converge in distribution as well, with the same centering and scaling, but a dif-
ferent limit distribution. This follows by combining the preceding results and the Poisson
approximation to the binomial distribution.
21.18 Theorem. Ifb;;1 (Xn(n) - an) G, then b;; 1 (Xn(n-k) - an) H for the distribution
function H(x) == G(x) L:7=o( -log G(x))i / i!.
314
Quantiles and Order Statistics
Proof. If Pn == F(a n + bnx), then npn -log G(x) for every x where G is continuous
(all x), by Lemma 21.12. Furthermore,
p(b;l(X n (n_k) - an) < x) == P(bin(n, Pn) < k).
This converges to the probability that a Poisson variable with mean - log G (x) is less than
or equal to k. (See problem 2.21.) .
By the same, but more complicated, arguments, the sample extremes can be seen to
converge jointly in distribution also, but we omit a discussion.
Any order statistic depends, by its definition, on all observations. Rowever, asymptot-
ically central and extreme order statistics depend on the observations in orthogonal ways
and become stochastically independent. One way to prove this is to note that central-order
statistics are asymptotically equivalent to means, and averages and extreme order statistics
are asymptotically independent, which is a result of interest on its own.
21.19 Lemma. Let g be a measurable function with F g == O and F g2 == 1, and sup-
pose that b;; 1 (Xn(n) -an) G for a nondegenerate distribution G. Then (n- 1 / 2 L:7=lg(X i ),
b;; 1 (Xn(n) - an)) (U, V) for independent random variables U and V with distributions
N (O, 1) and G.
Proof. Let Un == n- 1 / 2 L:7::f g(Xn(i)) and V n == b;; 1 (Xn(n) - an). Because Fg 2 < 00,
it follows that maxl::;i::;n Ig(Xi)1 == op(y'n). Renee n- 1 / 2 Ig(X n (n))1 O, whence the
distance between (Gng, V n ) and (Un, V n ) converges to zero in probability. It suffices to
show that (Un, V n ) (U, V). Suppose that we can show that, for every u,
p
Fn(u I V n ): == P(U n < u I V n ) cp(u).
Then, by the dominated-convergence theorem, EFn(u I V n )l{V n < v} == <I>(u)E1{V n <
v} + o (1), and hence the cumulative distribution function EFn (u I V n ) 1 {V n < v} of (Un, V n )
converges to cp(u)G(v).
The conditional distribution of Un given that V n == Un is the same as the distribution
of n- 1 / 2 L: X ni for i.i.d. random variables X n ,l, . . . , X n ,n-l distributed according to the
conditional distribution of g (X 1) given that XI < x n : == an + b n V n . These variables have
absolute mean
I I I I ( 2 - ) 1/2
gdF gdF g dF F(x n )
IEX n1 ! = J(-OO,x n ] = J(xn,OO) < J(xn,OO) .
F(x n ) F(x n ) - F(x n )
If v n v, then P(V n < vn) G(v) by the continuity of G, and, by Lemma 21.12, nF(x n ) ==
0(1) whenever G(v) > O. We conclude that y'nEX n1 O. Because we also have that
EX l Fg 2 and EX 11{IXnll > c:y'n} O for every c: > O, the Lindeberg-Feller
theorem yields that Fn (u I v n ) <I> (u). This implies Fn (u I V n ) <I> (u) by Theorem 18.11
or a direct argument. .
By taking linear combinations, we readily see from the preceding lemma that the em-
pirical process G n and b;; 1 (Xn(n) - an), if they converge, are asymptotically independent
as well. This independence carries over onto statistics whose asymptotic distribution can
Problems
315
be derived from the empirical process by the delta method, including centralorder statis-
tics Xn(kn/n) with knin == p + 0(11 -Jfi), because these are asymptotically equivalent to
averages .
Notes
For more results conceming the empirical quantile function, the books [28]and [134] are
good starting points. For results on extreme order statistics, see [66] or the book [90].
PROBLEMS
1. Suppose that Fn --+ F uniformly. Does this imply that Fn- 1 --+ F- 1 uniformly or pointwise? Give
a counterexample.
2. Show that the asymptotic lengths of the two types of asymptotic confidenee interval s for a quan-
tile, discussed in Example 21.8, are within o p (1/ ). Assume that the asymptotic variance of
the sample quantile (involving 1/ f o F- 1 (p)) can be estimated consistently.
3. Find the limit distribution of the median absolute deviation from the mean, med 1 i n IX i - X n I.
4. Find the limit distribution of the pth quantile of the absolute deviation from the median.
5. Prove that X n and X n (n-l) are asymptotically independent.
L-Statistics
In this chapter we prove the asymptotic normality of linear combinations
of order statistics, particularly those usedfor robust estimation or testing,
such as trimmed means. We present two methods: The projection method
presumes knowledge of Chapter 11 only; the second method is based on
the functional delta method of Chapter 20.
22.1 Introduction
Let X n (l), . . . , Xn(n) be the order statistics of a sample of real-valued random variables. A
linear combination of (transformed) order statistics, or L-statistic, is a statistic of the form
n
LCnia(Xn(i»)'
i=l
The coefficients Cni are a triangular array of constants and a is some fixed function. This
"score function" can without much loss of generality be taken equal to the identity function,
for an L-statistic with monotone function acan be viewed as a linear combination of the
order statistics of the variables a (X 1), . . . , a (X n), and an L-statistic with a function a of
bounded variation can be dealt with similarly, by splitting the L-statistic into two parts.
22.1 Example (Trimmed and Winsorized means). The simplest example of an L-statistic
is the sample mean. More interesting are the a-trimmed means t
1 n- LanJ
L Xn(i),
n - 2LanJ i=LanJ+l
and the a - Winsorized means
1 [ n-LanJ ]
- LanJ Xn(LanJ) + L Xn(i) + LanJ X n (n-LanJ+l) .
n i=LanJ+l
t The notation Lx J is used for the greatest integer that is less than or equal to x. Also r x l denotes the smallest
integer greater than or equal to x. For a natural number n and a real number O x n one has Ln - x J = n - r x l
and r n - x l = n - Lx J .
316
22.1 Introduction 317
Cauchy Laplace
(o o
('t')
I.[) I.[)
C\I
.q- o
C\I
('t') I.[)
T""
C\I
T""
I.[)
o
o o
o
0.0 0.2 0.4 0.0 0.2 0.4
normal logistic
o I.[)
('t')
I.[)
C\I
.q-
o
C\I
I.[) ('t')
T""
o
T""
C\I
I.[)
o
o
o
0.0 0.2 0.4 0.0 0.2 0.4
Figure 22.1. Asymptotic variance of the a-trimmed mean of a sample from a distribution F as
function of a for four distributions F.
The a-trimmed mean is the average of the middle (1 - 2a)-th fraction of the observations,
the a-Winsorized mean replaces the ath fractions of smallest and largest data by Xn(LanJ)
and X n (n-LanJ+l), respectively, and next takes the average. Both estimators were already
used in the early days of statistics as location estimators in situations in which the data were
suspected to contain outliers. Their properties were studied systematically in the context of
robust estimation in the 1960s and 1970s. The estimators were shown to have good properties
in situations in which the data follows a heavier tailed distribution than the normalone.
Figure 22.1 shows the asymptotic variances of the trimmed means as a function of a for
four distributions. (A formula for the asymptotic variance is given in Example 22.11.) The
four graphs suggest that 10% to 15% trimming may give an improvement over the sample
mean in some cases and does not cost much even for the normal distribution. D
22.2 Example (Ranges). Two estimators of dispersion are the interquartile range
X n (l3nf4l) - X n (lnf4l) and the range Xn(n) - X n (1). Of these, the range does not have a
normallimit distribution and is not within the scope of the results of this chapter. D
We present two methods to prove the asymptotic normality of L-statistics. The first
method is based on the Hajek projection; the second uses the delta method. The second
method is preferable in that it applies to more general statistics, but it necessitates the study
of empirical processes and does not cover the simplest L-statistic: the sample mean.
318
L-Statistics
22.2 Hajek Projection
The Hajek projection of a general statistic is discussed in section 11.3. Because a projection
is linear and an L-statistic is line ar in the order statistics, the Hajek projection of an L-statistic
can be found from the Hajek projections of the individual order statisties. Up to centering
at mean zero, these are the sums of the conditional expectations E(Xn(ž) I X k ) over k. Some
thought shows that the conditional distribution of Xn(ž) given Xk is given by
I P(Xn-l(ž) < y)
P(Xn(ž) < yI X k == x) ==
P(Xn-l(ž-l) < y)
if y < x,
if y > x.
This is correct for the extreme cases i == 1 and i == n provided that we de fine X n -1 (O) == - 00
and Xn-l(n) == 00. Thus, we obtain, by the partial integration formula for an expectation,
for x > O,
E(Xn(i) I X k = x) = iX P(X n - 1 (i) > y) dy + 1 00 P(X n - 1 (i-1) > y) dy
- i: P(X n - 1 (i) < y) dy
= -1 00 (P(X n - 1 (i) > y) - P(X n - 1 (i-1) > y)) dy + EX n - 1 (i).
The second expression is valid for x < O as well, as can be seen by a similar argument.
Because Xn-l(i-I) < Xn-l(i), the difference between the two probabilities in thelastintegral
is equal to the probability of the event {Xn-l(i-I) < y < Xn-l(i)}. This is precisely the
probability that a binomial (n -1, F (y) ) - variable is equal to i-I. If we write this probability
as B n - 1 ,F(y) (i - 1), then the Hajek projectionXn(i) of Xn(i) satisfies, with JF n the empirical
distribution function of XI, . . . , X n,
n 1 00
Xn(i) - E 1(i) == - L B n - 1 ,F(y)(i - 1) dy + C n
k=l Xk
= - f n(IF n - F)(y) B n - 1 ,F(y)(i - 1) dy.
For the projection of the L-statistic Tn == I: 7= 1 Cni Xn(i) we find
Tn - ETn = - f n(IFn - F)(y) tCni B n - 1 ,F(y) (i - 1) dy.
i=l
Under some conditions on the coefficients Cni, this sum (divided by .Jfi) is asymptotically
normal by the central limit theorem. Furthermore, the projection Tn can be shown to be
asymptotically equivalent to the L-statistic Tn by Theorem 11.2. Sufficient conditions on
the Cni can take a simple appearance for coefficients that are "generated" by a function C/J
as in (13.4).
22.3 Theorem. Supposethat EXi < ooandthatcni ==c/J(ij(n+1))Joraboundedfunction
C/J that is continuous at F (y )Jor Lebesgue almost-every y. Then the sequence n- 1 / 2 (Tn - ETn)
22.2 Hajek Projection
319
converges in distribution to a normal distribution with mean zero and variance
a 2 (ef>, F) = ff ef> ( F(x) ) ef> ( F(y)) (F(x !\ y) - F(x )F(y)) dx dy.
ProoJ. Define functions e (y) == ep ( F (y)) and
n ( B +1 )
en(y) == LCni B n - 1 ,F(y) (i - 1) == Eep n ,
i=l n + 1
for Bn binomially distributed with parameters (n - 1, F (y) ). By the law of large numbers
(Bn + l)/(n + 1) F(y). Because ep is bounded, en(y) -+ e(y) for every y such that ep is
continuous at F(y), by the dominated-convergence theorem. By assumption, this includes
almost every y.
By Theorem 11.2, the sequence n- 1 / 2 (Tn - Tn) converges in second mean to zero if the
variances ofn- 1 / 2 T n and n- 1 / 2 Tn converge to the samenumber. Becausen- 1 / 2 (Tn - ETn) ==
- J CG n (y) en (y) dy, the second variance is easily computed to be
var Tn = ff (F(x !\ y) - F(x)F(y)) en (x)en (y) dx dy.
This converges to a 2 (ep, F) by the dominated -convergence theorem. The variance of n-I /2 Tn
can be written in the form
lIn n ff
-var Tn == - LLCniCnj cov(Xn(i), Xn(j)) == Rn(x, y) dx dy,
n n. 1 . 1
l= J=
where, because cov(X, Y) == JJ cov({X < x}, {Y < y}) dx dy for any pair of variables
(X, Y),
In n ( ' )( . )
Rn(x, y) = - LI) 1 ef> ] cov({Xn(i) < x}, {Xn(j) < y}).
n i=l j=l n + 1 n + 1
Because the order statistics are positively correlated, all covariances in the double sum are
nonnegative. Furthermore,
1 n n
- LL cov({Xn(i) < x}, {Xn(j) < y}) == cov(CGn(x), CGn(y))
n. 1 . 1
l= J=
== (F (x /\ y) - F (x ) F (y ) ) .
For pairs (i, j) such that i nF(x) and j nF(y), the coefficient of the covariance
is approximately e(x )e(y) by the continuity of ep. The covariances corresponding to other
pairs (i, j) are negligible. Indeed, for i > nF(x) + nSn,
o < cov({Xn(i) < x}, {Xn(j) < y}) < 2P(X n (i) < x)
< 2P(bin(n, F(x)) > nF(x) + nSn)
< 2 exp -2ns ,
320
L-Statistics
by Hoeffding's inequality.t Thus, because ep is bounded, the term s with i > nF(x) + nSn
contribute exponentially little as 8n O not too fast (e.g., S == n- I / 2 ). A similar argument
applies to the terms with i < nF(x) - nSn or li - nF(y) I > nSn. Conclude that, for every
(x, y) such that ep is continuous at both F (x) and F (y ),
Rn(x, y) e(x)e(y)(F(x 1\ y) - F(x)F(y)).
Finally, we apply the dominated convergence theorem to see that the double integral of this
expression, which is equal to n- I var Tn, converges to a 2 (ep, F).
This concludes the proof that Tn and Tn are asymptotically equivalent. To show that
the sequence n- I / 2 (Tn - ETn) is asymptotically normal, define Sn == - fGn(y)e(y)dy.
Then, by the same argument s as before, n- I var(Sn - Tn) O. Furthermore, the sequence
n- I / 2 Sn is asymptotically normal by the central limit theorem. .
22.3 Delta Method
The order statistics of a sample XI, . . . , X n can be expressed in their empirical distribution
1F n, or rather the empirical quantile function, through
1F;;-I(S) == Xn(lsnl) == Xn(i),
i-I l
for < s < -.
n n
Consequently, an L-statistic can be expressed in the empirical distribution function as well.
Given a fixed function a and a fixed signed measure K on (O, 1) +, consider the function
ifJ(F) = 1] a(F-])dK.
View ep as a map from the set of distribution functions into IR. Clearly,
n ( . 1 . ]
ifJ(JF n) = LK I - , a(Xn(i))'
i=l n n
(22.4 )
The right side is an L-statistic with coefficients Cni == K( (1 - l)ln, lin]. Not all possible
arrays of coefficients Cni can be "generated" through a measure K in this manner. However,
most L-statistics of interest are almost of the form (22.4), so that not much generality is lost
by assuming this structure. An advantage is simplicity in the formulation of the asymptotic
properties of the statistics, which can be derived with the help of the von Mises method.
More importantly, the function ep (F) can also be applied to other estimators besides 1F n.
The results of this section yield their asymptotic normality in general.
22.5 Example. The a-trimmed mean corresponds to the uniform distribution K on the
interval (a, 1 - a) and a the identity function. More precisely, the L-statistic generated by
t See for example, the appendix of [117]. This inequality gives more than needed. For instance, it also works to
apply Markov's inequality for fourth moments.
+ A signed measure is a difference K = K 1 - K 2 of two finite measures K 1 and K 2.
22.3 Delta Method
321
this measure is
1 l 1 - a 1 [
IF;l(s) ds == (Ianl - an)Xn(fanl)
1 - 2a n - 2an
a
n-ianl ]
+ L Xn(i) + (I an l - an )X n (n-ianl+1) .
i=ianl+1
Except for the slightly different weight factor and the treatment of the two extremes in
the averages, this agrees with the a-trimmed mean as introduced before. Because Xn(k n )
converges in probability to F- 1 (p) if knin -+ p and (n - 2 LanJ)/(n - 2an) == 1 + OCI/n),
the difference between the two versions of the trimmed mean can be seen to be O p (1/ n).
For the purpose of this chapter this is negligible.
The a-Winsorized mean corresponds to the measure K that is the sum of Lebesgue
measure on (a, 1 - a) and the discrete measure with pointmasses of size a at each of the
points a and 1 - a. Again, the difference between the estimator generated by this K and
the Winsorized mean is negligible.
The interquartile range corresponds to the discrete, signed measure K that has point-
masses of sizes 1 and -1 at the points 1/4 and 3/4, respectively. O
Before giving a proof of asymptotic normality, we derive the influence function of an
(empirical) L-statistic in an informal way. If Ft == (1 - t)F + t8x, then, by definition, the
influence function is the derivative of the map t ep (Ft) at t == O. Provided a and K are
sufficiently regular,
:t 11 a(Ft- 1 ) dK = 11 a'(Ft- 1 ) [ :t Ft- 1 ] dK.
Here the expression within square brackets if evaluated at t == O is the influence function of
the quantile function and is derived in Example 20.5. Substituting the representation given
there, we see that the influence function of the L-function ep(F) == J a(F- 1 ) dK takes the
form
1 1 1 ( F- 1 (u )) U
rh' (8 - F) == - a' ( F- 1 (u) ) [x,oo) - dK (u)
O/F x o f(F-1(u))
== - f a'(y) l[x,oo)(y) - F(y) dK o F(y),
f(y)
(22.6)
The second equality follows by (a generalization of) the quantile transformation.
An alternative derivation of the influence function starts with rewriting ep (F) in the form
ep(F) == f adK o F == [ (K o F)_da - [ (K o F)_da.
1(0,00) 1( -00,0]
(22.7)
Here K o F (x) == K o F (00) - K o F (x) and the partial integration can be justified for
a a function of bounded variation with aCO) == O (see problem 22.6; the assumption that
aCO) == O simplifies the formula, and is made for convenience). This formula for ep (F)
suggests as influence function
cjJ (8x - F) = - f K' (F(y)) (1 [x,oo) (y) - F(y)) da(y),
(22.8)
322
L-Statistics
Under appropriate conditions each of the two formulas (22.6) and (22.8) for the infiuence
function is valid. However, already for the defining expressions to make sense very different
conditions are needed. Informally, for equation (22.6) it is necessary that a and F be
differentiable with apositive derivative for F, (22.8) requires that K be differentiable. For
this reason both expressions are valuable, and they yield non overlapping results.
Corresponding to the two derivations of the infiuence function, there are two basic
approaches towards proving asymptotic normality of L-statistics by the delta method, valid
under different sets of conditions. Roughly, one approach requires that F and a be smooth,
and the other that K be smooth.
The simplest method is to view the L-statistic as a function of the empirical quantile
function, through the map IF; 1 1-+ f a o IF; 1 dK, and next apply the functional delta method
to the map Q 1-+ f a o Q dK. The asymptotic normality of the empirical quantile function
is obtained in Chapter 21.
22.9 Lemma. Let a : IR 1-+ IR be continuously differentiable with a bounded derivative.
Let K be a signed measure on the interval (a, tJ) c (O, 1). Then the map Q 1-+ f a(Q) dK
from £OO(a, tJ) to IR is Hadamard-differentiable at every Q. The derivative is the map
H 1-+ f a'(Q) H dK.
Proof. Let Ht H in the uniform norm. Consider the difference
f a(Q + tI;) - a(Q) - a'(Q) H dK.
The integrand converges to zero everywhere and is bounded uniformlyby Ila' 1100 (II Ht 1100+
II H II (0)' Thus the integral converges to zero by the dominated-convergence theorem. .
If the underlying distribution has unbounded support, the n its quantile function is un-
bounded on the domain (O, 1), and no estimator can converge in ,eOO(O, 1). Then the pre-
ceding lemma can apply only to generating measures K with support (a, tJ) strictly within
(O, 1). Fortunately, such generating measures are the most interesting ones, as they yield
bounded influence functions and hence robust L-statistics.
Amore serious 1imitation of using the preceding lemma is that it could require unnec-
essary smoothness conditions on the distribution of the observations. For instance, the
empirical quanti1e process converges in distribution in £00 (a, tJ) only if the underlying dis-
tribution has apositive density between its a- and tJ-quanti1es. This is true for most standard
distributions, but unnecessary for the asymptotic normality of empirical L-statistics gen-
erated by smooth measures K. Thus we present a second lemma that applies to smooth
measures K and does not require that F be smooth. Let DF[ -00,00] be the set of all
distribution functions.
22.10 Lemma. Let a : IR 1-+ IR be ofbounded variation on bounded intervals with f (a+ +
a-) dlK o FI < 00 and aCO) == O. Let K be a signed measure on (O, 1) whose distribution
function K is differentiable at F (x) for a almost-every x and satisfies I K (u + h) - K (u) I <
M(u)hfor every sufficiently smalllhl, and somefunction M such that f M(F_) dlal < 00.
Then the map F 1-+ f a o F- I dK from DF[-oo, 00] C D[-oo, 00] to JR is Hadamard-
differentiable at F, with derivative H 1-+ - f (K' o F _) H da.
22.4 L-Estimators for Location
323
Proof. First rewrite the function in the form (22.7). Suppose that Ht ---+ H uniformly and
set Ft == F + tHt. By continuity of K, (K o F)_ == K(F_). Because K o F(oo) == K(1)
for all F, the difference cjJ(Ft) - cjJ(F) can be rewritten as - J(K o Ft- - K o F_)da.
Consider the integral
J K(F_ + tHt-) - K(F_) ,
- K (F_)H dlal.
t
The integrand converges a-almost everywhere to zero and is bounded by M(F_) (II Ht 1100 +
IIHlloo) < M(F_)(21IHlloo + 1), for small t. Thus, the lemma follows by the dominated-
convergence theorem. .
Because the two lemmas apply to nonoverlapping situations, it is worthwhile to combine
the two approaches. A given generating measure K can be split in its discrete and continuous
part. The corresponding two parts of the L-statistic can next be shown to be asymptotically
linear by application of the two lemmas. Their sum is asymptotically linear as well and
hence asymptotically norma!.
22.11 Example (Trimmed mean). The cumulative distribution function K of the uniform
distribution on (a, 1 - a) is uniformly Lipschitz and fails to be differentiable only at the
points ex and 1 - a. Thus, the trimmed-mean function is Hadamard-differentiable at every
F such that the set {x : F(x) == a, or 1 - a} has Lebesgue measure zero. (We assume that
a > O.) In other words, F should nothave flats atheight a or 1-a. For such F the trimmed
mean is asymptotically normal with asymptotic influence function - J:- a (1x y - F (y)) dy
(see (22.8)), and asymptotic variance
1 p-I (1-a) l p-I (1-a)
(F(x /\ y) - F(x) F(y)) dx dy.
p-I (a) p-I (a)
Figure 22.1 shows this number as a function of a for a number of distributions. D
22.12 Example (Winsorized mean). The generating measure of the Winsorized mean is
the sum of a discrete measure on the two points a and 1 - a, and Lebesgue measure on
the interval (a, 1 - a). The Winsorized mean itself can be decomposed correspondingly.
Suppose that the underlying distribution function F has apositive derivative at the points
F- I (a) and F- I (1 - a). Then the first part of the decomposition is asymptotically linear
in view of Lemma 22.9 and Lemma 21.3, the second part is asymptotically linear by
Lemma 22.10 and Theorem 19.3. Combined, this yields the asymptotic linearity of the
Winsorized mean and hence its asymptotic normality. D
22.4 L-Estimators for Location
The a - trimmed mean and the a - Winsorized mean were invented as estimators for location.
The question in this section is whether there are still other attractive location estimators
within the class of L-statistics.
One possible method of generating L-estimators for location is to find the best L-
estimators for given location families {f (x - e) : e E IR}, in which f is some fixed density.
For instance, for the f equal to the normal shape this leads to the sample mean.
324
L-Statistics
According to Chapter 8, an estimator sequence Tn is asymptotically optimal for estimat-
ing the location of a density with finite Fisher information I f if
1 n 1 f'
-Jfi(Tn - e) == --2: --(Xi - e) + opel).
-Jfii=lIff
Comparison with equation (22.8) for the influence function of an L-statistic shows that the
choices of generating measure K and transformation a such that
( 1 f' ) ,
K'(F(x - e)) a'ex) == - --ex - e)
If f
lead to an L-statistic with the optimal asymptotic influence function. This can be accom-
modated by setting a(x) - x and
K'(u) == _ ( fl ) ' (p-l(u)).
If f
The class of L-statistics is apparently large enough to contain an asymptotically efficient
estimator sequence for the location parameter of any smooth shape. The L-statistics are
not as simplistic as they may seem at first.
Notes
This chapter gives only a few of the many results available on L-statistics. For instance,
the results on Hadamard differentiability can be refined by using a weighted uniform norm
combined with convergence of the weighted empirical process. This allows greater weights
for the extreme-order statistics. For further resuits and references, see [74], [134], and
[136] .
PROBLEMS
1. Find a formula for the asymptotic variance of the Winsorized mean.
2. Let T(F) == f F- 1 (u) k(u) du.
(i) Show that T (F) == O for every distribution F that is symmetric about zero if and only if
k is symmetric about 1/2.
(ii) Show that T (F) is location equivariant if and only if f k(u) du == 1.
(iii) Show that "efficient" L-statistics obtained from symmetric densities possess both prop-
erties (i) and (ii).
3. Let XI, . . . , X n be arandom sample from a continuous distribution function. Show that con-
ditionally on (Xn(k), Xn(l)) == (x, y), the variables X n (k+l),..., X n (l-l) are distributed as the
order statistics of arandom sample of size I - k - 1 from the conditional distribution of XI
given that x ::s XI ::s y. How can you use this to study the properties of trimmed means?
4. Find an optimal L-statistic for estimating the location in the logistic and Laplace location
families.
5. Does there exist a distribution for which the trimmed mean is asymptotically optimal for esti-
mating location?
Problems
325
6. (Parti al Integration.) If a : JR f--+ JR is right-continuous and nondecreasing with a (O) == O, and
b: JR f--+ JR is right-continuous, nondecreasing and bounded, then
J adb== r (b(OO)-b_)da+ j (b(-oo)-b_)da.
1(0,00) (-00,0]
Prove this. If a is also bounded, then the righthand side can be written more succinctly as
abl oo - f b- da. (Substitute a(x) == f(o,x] da for x > O and a(x) == - f(x,o] da for x :s O
into the left side of the equation, and use Fubini's theorem separately on the integral over the
positive and negative part of the real line.)
23
Bootstrap
This chapter investigates the asymptotic properties of bootstrap estima-
tors for distributions and confidenee intervals. The consistency of the
bootstrap for the sample mean implies the consistency for many other
statistics by the delta method. A similar result is valid with the empirical
process.
23.1 Introduction
In most estimation problem s it is important to give an indication of the precision of a given
estimate. A simple method is to provide an estimate of the bias and variance of the estimator;
more accurate is a confidence interval for the parameter. In this chapter we concentrate on
bootstrap confidence interval s and, more generally, discuss the bootstrap as a method of
estimating the distribution of a given statistic.
Let e be an estimator of some parameter e attached to the distribution P of the obser-
vations. The distribution of the difference e - e contains all the information needed for
assessing the precision of e. In particular, if a is the upper a -quantile of the distribution
of (e - e)j8-, then
p (e - f3 8- < () < e - I-a 8- I P) > 1 - fJ - a.
Here 8- may be arbitrary, but it is typically an estimate of the standard deviation of e. lt
follows that the interval [e - f3 8- , e - I-a 8-] is a confidence interval of level 1 - tJ - a.
Unfortunately, in most situations the quantiles and the distribution of e - () depend on the
unknown distribution P of the observations and cannot be us ed to assess the performance
of e. They must be replaced by estimators.
If the sequence (e - e) j 8- tends in distribution to a standard normal variable, then the
normal N (O, 8- 2 ) -distribution can be used as an estimator of the distribution of e - () ,
and we can substitute the standard normal quantiles Za for the quantiles a' The weak
convergence implies that the interval [e - Z tJ 8- , e - Z I-a 8-] is a confidence interval of
asymptotic level 1 - a - tJ.
Bootstrap procedures yield an alternative. They are based on an estimate P of the un-
derlying distribution P of the observations. The distribution of (e - ())j8- under P can, in
principle, be written as a function of P. The bootstrap estimator for this distribution is the
"plug-in" estimator obtained by substituting P for P in this function. Bootstrap estimatars
326
23.1 Introduction
327
for quantiles, and next confidence intervals, are obtained from the bootstrap estimator for
the distribution.
The following type of notation is customary. Let e* and a* be computed from (hypo-
thetic) observations obtained according to P in the same way e and a are computed from
the true observations with distribution P. If e is related to P in the same way e is related to
P, then the bootstrap estimator for the distribution of (e - e) / a under P is the distribution
of (e* - e) / a* under P. The latter is evaluated given the original observations, that is, for
a fixed realization of P .
A bootstrap estimator for a quantile ct of (e - 8) / a is a quantile of the distribution of
(e * - e) / a * under P. This is the smallest value x == ct that satisfies the inequality
p( e*O' e < X 1 P» 1-a.
The notation P(. I P) indicates that the distribution of (e*, a*) must be evaluated assum-
ing that the observations are sampled according to P given the original observations. In
particular, in the preceding display e is to be considered nonrandom. The left side of the
preceding display is a function of the original observations, whence the same is true for ct'
If P is close to the true underlying distribution P, then the bootstrap quantiles should be
close to the true quantiles, whence it should be true that
(23.1 )
( e - 8 )
P o' < €a I p 1 - a.
In this chapter we show that this approximation is valid in an asymptotic sense: The
probability on the left converges to 1 - ex as the number of observations tends to infinity.
Thus, the bootstrap confidence interval
'" " '" '" { " e-8 "' }
[8 - f3a, 8 - l-cta] == 8: l-ct < a < f3
possesses asymptotic confidence levelI - ex - tJ.
The statistic a is typically chosen equal to an estimator of the (asymptotic) standard
deviation of e. The resulting bootstrap method is known as the percentile t -method, in view
of the fact that it is based on estimating quantiles of the "studentized" statistic (e - 8) / a.
(The notion of a t -statistic is used here in an abstract manner to denote a centered statistic
divided by a scale estimate; in general, there is no relationship with Student's t-distribution
from normal theory.) Asimpler method is to choose a independent of the data. If we
choose a == a * == 1, then the bootstrap quantiles ct are the quantiles of the centered statistic
e* - e. This is known as the percentile method. Both methods yield asymptotically correct
confidence levels, although the percentile t-method is generally more accurate.
A third method, Efron 's percentile method, proposes the confidence interval [ 1-f3' ct]
for ct equal to the upper ex -quantile of e *: the smallest value x == ct such that
p(e* < xi P) > I-ex.
Thus, ct results from "bootstrapping" e, while ct is the product of bootstrapping (e -
8) / a. These quantiles are related, and Efron's percentile interval can be reexpressed in the
quantiles ct of e * - e (employed by the percentile method with a == 1) as
[ 1-f3' ct] == [e + 1-f3' e + ct].
328
Bootstrap
The logical justification for this interval is less strong than for the interval s based on boot-
strapping 8 - e, but it appears to work well. The two types of interval s coincide in the case
that the conditional distribution of 8 * - 8 is symmetric about zero. We shall see that the
difference is asymptotically negligible if 8 * - 8 converges to a normal distribution.
Efron' s percentile interval is the only one among the three interval s that is invariant
under monotone transformations. For instance, if setting a confidence interval for the cor-
relation coefficient, the sample correlation coefficient might be transformed by Fisher' s
transformation before carrying out the bootstrap scheme. N ext, the confidence interval
for the transformed correlation can be transformed back into a confidence interval for the
correlation coefficient. This operation would have no effect on Efron' s percentile interval
but can improve the other interval s considerably, in view of the skewness of the statistic. In
this sense Efron's method automatically "finds" useful (stabilizing) transformations. The
fact that it does not become better through transformations of course does not imply that it
is good, but the invariance appears desirable.
Several of the elements of the bootstrap scheme are still unspecified. The missing prob-
ability ex + fJ can be distributed over the two tails of the confidence interval in several ways.
In many situations equal-tailed confidence intervals, corresponding to the choice ex == fJ,
are reasonable. In general, these do not have 8 exactly as the midpoint of the interval. An
alternative is the interval
[8 - +f30-, 8 + +f30-],
with equal to the upper ex -quantile of 18* - 8 1/ {j- *. A further possibility is to choose ex
and fJ under the side condition that the difference f3 - l-a' which is proportional to the
length of the confidence interval, is minimal.
More interesting is the choice of the estimator P for the underlying distribution. If the
original observations are arandom sample XI, . . . , X n from a probability distribution P,
then one candidate is the empirical distribution JP n == n-I L o Xi of the observations, leading
to the empirical bootstrap. Generating arandom sample from the empirical distribution
amounts to resampling with replacement from the set {X 1, . . . , X n} of original observations.
The name "bootstrap" derives from this resampling procedure, which might be surprising
at first, because the observations are "sampled twice." If we view the bootstrap as a
nonparametric plug-in estimator, we see that there is nothing peculiar about resampling.
We shall be mostly concerned with the empirical bootstrap, even though there are many
other possibilities. If the observations are thought to follow a specified parametric model,
then it is more reasonable to set P equal to P{) for a given estimator 8. This is what one
would have done in the first place, but it is called the parametric bootstrap within the
present context. That the bootstrapping methodology is far from obvious is clear from
the fact that the literature also considers the exchangeable, the Bayesian, the smoothed, and
the wild bootstrap, as well as several schemes for bootstrap corrections. Even "resampling"
can be carried out differently, for instance, by sampling fewer than n variables, or without
replacement.
It is almost never possible to calculate the bootstrap quantiles a numerically. In practice,
these estimators are approximated by a simulation procedure. A large number of indepen-
dent bootstrap samples Xr, . . . , X are generated according to the estimated distribution
P. Each sample gives rise to a bootstrap value (8 * - 8) / O- * of the standardized statistic.
Finally, the bootstrap quantiles €a are estimated by the empirical quantiles of these bootstrap
23.2 Consistency
329
values. This simulation scheme always produces an additional Crandom) error in the cover-
age probability of the resulting confidence interval. In principle, by using a sufficiently large
number of bootstrap samples, possibly combined with an efficient method of simulation,
this error can be made arbitrarily small. Therefore the additional error is usually ignored in
the theory of the bootstrap procedure. This chapter follows this custom and concerns the
"exact" distribution and quantiles of (e* - e) /0-*, without taking a simulation error into
account.
23.2 Consistency
A confidence interval [en, 1, e n ,2] is Cconservatively) asymptotically consistent at leveli -
ex - f3 if, for every possible P,
liminf PCe n ,l < e < e n ,21 P) > 1 - ex - f3.
n ---+ 00
The consistency of a bootstrap confidence interval is closely related to the consistency of
the bootstrap estimator of the distribution of (en - e) /o-n. The latter is best defined relative
to a metric on the collection of possible laws of the estimator. Call the bootstrap estimator
for the distribution consistent relative to the Kolmogorov-Smirnov distance if
( en - e ) ( e* - en ,, ) p
sup P " < xi P -p n "* < xi Pn ---+ O.
x an an
It is not a great loss of generality to assume that the sequence Ce n - e) /o-n converges in
distribution to a continuous distribution function F Cin our examples cI». Then consistency
relative to the Kolmogorov-Smirnov distance is equivalent to the requirements, for every x,
( e - e )
p na-n < X I p -+ F(x),
( e* - en " ) p
p n a-; < x I p n -+ F(x).
C23.2)
CSee Problem 23.1.) This type of consistency implies the asymptotic consistency of confi-
dence intervals.
23.3 Lemma. Suppose that (en - e) / o-n T, and that (e: - en) / 0-; T given the orig-
inal observations, in probability, for arandom variable T with a continuous distribution
function. Then the bootstrap confidenee intervals [en - €n,f3o-n, en - €n,l- a o-n] are asymp-
totically consistent at leveli - ex - f3. If the conditions hold for nonrandom O-n == 0-;,
and T is symmetrically distributed about zero, then the same is true for Efron 's percentile
intervals.
Prool Every subsequence has a further subsequence along which the sequence Ce: -
en) / a-: converges weakly to T, conditionally, almost surely. For simplicity, assume that
the whole sequence converges almost surely; otherwise, argue along subsequences.
If a sequence of distribution functions Fn converges weakly to a distribution function
F, then the corresponding quantile functions Fn- 1 converge to the quantile function F- 1 at
every continuity point Csee Lemma 21.2). Apply this to the Crandom) distribution functions
Fn of Ce: - en) /0- , : and a continuity point 1 - ex of the quantile function F- 1 of T to conclude
330
Bootstrap
that n,a == -1 (1 - a) converges almost surely to p-I (1 - a). By Slutsky's lemma, the
sequence (en - ())/o-n - n,a converges weakly to T - p-I (1 - a). Thus
A A A ( en - () A ) ( -1 )
P(() > ()n - (Jn n,a) == p o-n < n,a I p -* P T < P (1 - a) == 1 - a.
This argument applies to all except at most countably many a. Because both the left and
the right sides of the preceding display are monotone functions of a and the right side
is continuous, it must be valid for every a. The consistency of the bootstrap confidence
interval follows.
Efron's percentile interval is the interval [ n,l-.B' n,a]' where n,a == en + n,a]' By the
preceding argument,
P(() > n,l-.B) == P(e n - () < - n,l-.B I P) --+ P(T < _F-l( )) == 1 - .
The last equality follows by the symmetry of T. The consistency follows. .
From now on we consider the empirical bootstrap; that is, P n == 1Fn is the empirical
distribution of arandom sample XI, . . . , Xn. We shalI establish (23.2) for a large class of
statistics, with P the normal distribution. Our method is first to prave the consistency for en
equal to the sample mean and next to show that the consistency is retained under application
of the delta method. Combining these resuits, we obtain the consistency of many bootstrap
procedures, for instance for setting confidence interval s for the correlation coefficient.
In view of Slutsky's lemma, weak convergence of the centered sequence -J1i(e n - ())
combined with convergence in probability of O- n I -J1i yields the weak convergence of the
studentized statistics (en - ())/o-n. An analogous statement is true for the bootstrap statis-
tic, for which the convergence in probability of 0-: I -J1i must be shown conditionally on
the original observations. Establishing (conditional) consistency of o-nl -J1i and 0-; / -J1i is
usually not hard. Therefore, we restrict ourselves to studying the nonstudentized statisties.
Let X n be the mean of a sample of n random vectors from a distribution with finite mean
vector f.-L and covariance matrix b. According to the multivariate central limit theorem, the
sequence -J1i( X n - f.-L) is asymptotically normal N(G, b)-distributed. We wish to show the
same for -J1i( X - X n ), in which X is the average of n observations from 1Fn, that is, of n
values resampled from the set of original observations {XI, . . . , X n} with replacement.
23.4 Theorem (Sample mean). Let XI, X 2 , . . . be i.i.d. random vectors with mean f.-L
and covariance matrix b. Then conditionally on XI, X 2, . . . , for almost every sequence
X 1 ,X 2 ,...,
( X - X n ) N(G, b).
Proof. For a fixed sequence XI, X 2 , . . . , the variable X is the average of n observa-
tions X ' . . . , X sampled from the empirical distribution 1F n. The (conditional) mean and
covariance matrix of these observations are
n 1 _
E(X; I Pn) == L -Xi == X n ,
. I n
1=
* - * - T ) 1 - - T
E((X i - Xn)(X i - X n ) I JP n == -(Xi - Xn)(X i - X n )
. I n
1=
--T
== XnXI - XnX n .
23.2 Consistency
331
By the strong law of large numbers, the conditional covariance converges to h for almost
every sequence XI, X 2 , . . . .
The asymptotic distribution of X can be established by the central limit theorem. Be-
cause the observations X , . . . , X are sampled from a different distribution n for every
n, a central limit theorem for a triangular array is necessary. The Lindeberg central limit
theorem, Theorem 2.27, is appropriate. It suffices to show that, for every E > O,
1 n
EIIX;11 2 1{IIX;11 > E0l} == - LIIX i I1 2 1{IIX i ll > E0l} O.
n. 1
l=
The left side is smaller than n- I L:7=IIIX i Il 2 1{IIX i ll > M} as soon as E0l > M. By
the strong law of large numbers, the latter average converges to EllX i 11 2 1 {II Xi II > M} for
almost every sequence XI, X 2, . . . . For sufficiently large M, this expression is arbitrarily
smal!. Conclude that the limit superior of the left side of the preceding display is smaller
than any number 1] > O almost surely and hence the left side converges to zero for almost
every sequence XI, X 2 , . . .. .
Assume that en is a statistic, and that ep is a given differentiable map. If the sequence
0l (Đ n - e) converges in distribution, then so does the sequence 0l (ep (en) - ep (e) ), by the
delta method. The bootstrap estimator for the distribution of ep (en) - ep (e) is ep (e;) - ep (en)'
If the bootstrap is consistent for estimating the distribution of en - e, then it is also consistent
for estimating the distribution of ep (en) - ep (e).
23.5 Theorem (Delta methodfor bootstrap). Let ep : JRk 1---+ JRm be a measurabIe map
defined and continuously differentiable in a neighborhood of e. Let en be random vectors
taking their vaIues in the domain of ep that converge aImost sureIy to e. If..Jii (en - e) T,
and -Jn(e; - en) T conditionally aImost sureIy, then both ..Jii(ep(e n ) - ep (e)) ep (T)
and -Jn (ep (e;) - ep (en) ) ep (T), conditionally aImost sure Iy.
Proof. By the mean value theorem, the difference ep (e;) -ep (en) can be written as ep (e;-
A ,.., A A n
en) for a point en between e; and en, ifthe latter two points are in the ball around e in which
ep is continuously differentiable. By the continuity of the derivative, there exists for every
TJ > O a constant 8 > O such that lIep /h - ep h II < 1] IIh II for every h and every lIe ' - e II < 8.
If n is sufficiently large, 8 sufficiently small, ..Jiille; - en II < M, and lien - eli < 8, then
Rn: = 110l(<fJ(e:) - <fJ(e n )) - <fJ 0l(e: - en) II
I I I C "'* '" I
== (epe n - ep())y n (en - en) < 1]M.
Fix a number E > O and a large number M. For 1] sufficiently small to ensure that TJ M < E,
P(Rn > El Pn) < p(0llle; - en II > M or lien - eli> 81 Pn).
Because en converges almost surely to e, the right side converges almost surely to p(1I T II >
M) for every continuity point M of II T II. This can be made arbitrarily small by choice of
M. Conclude that the left side converges to O almost surely. The theorem follows by an
application of Slutsky's lemma. .
23.6 Example (Sample variance). The (biased) sample variance S == n-I L:7=I (Xi -
Xn )2 equals ep ( X n , X; ) for the map ep (x, y) == y - x 2 . The empirical bootstrap is consistent
332
Bootstrap
for estimation of the distribution of ( X n , X ) - (al, (2), by Theorem 23.4, provided
that the fourth moment of the underlying distribution is finite. The delta method shows
that the empirical bootstrap is consistent for estimating the distribution of S;; - a 2 in
that
sup P( .Jn(S - 0- 2 ) < x I p) - P( .Jn(S 2 - S ) < x I JP>n) o.
x
The asymptotic variance of S;; can be estimated by S (kn + 2), in which kn is the sample
kurtosis. The law of large numbers shows that this estimator is asymptotically consistent.
The bootstrap version of this estimator can be shown to be consistent given almost every
sequence of the original observations. Thus, the consis tency of the empirical bootstrap
extends to the studentized statistic (S;; - a 2 ) I S, ,J kn + 1. D
*23.2.1 Empirical Bootstrap
In this section we follow the same method as previously, but we replace the sample mean
by the empirical distribution and the delta method by the functional delta method. This is
more involved, but more flexible, and yields, for instance, the consistency of the bootstrap
of the sample median.
Let IP n be the empirical distribution of arandom sample XI, . . . , X n from a distribution
p on a measurable space (X, A), and let F be a Donsker class of measurable functions
f : X 1---+ IR, as defined in Chapter 19. Given the sample values, let Xr, . . . , X be arandom
sample from JP n. The bootstrap empirical distribution is the empirical measure JP
n-I I:7 = 18 x7' and the bootstrap empirical process G is
* C* 1
G n == v n(JP n - JP n ) == vn (Mni - 1) 8xi'
n. 1
1=
in which M ni is the number of times that Xi is "redrawn" from {XI, . . . , X n} to form
Xr, . . . , X . By construction, the vector of counts (M n1 ,..., M nn ) is independent of
XI, ..., X n and multinomially distributed with parameters n and (probabilities) lin, ...,
lin.
If the class F has a finite envelope function F, then both the empirical process G n and
the bootstrap process G can be viewed as maps into the space £oo(F). The analogue of
Theorem 23.4 is that the sequence CG converges in £00 (F) conditionally in distribution to
the same limit as the sequence G n , a tight Brownian bridge process G p. To give a precise
meaning to "conditional weak convergence" in £ 00 (F), we use the bounded Lipschitz metric.
It can be shown that a sequence of random elements in £00 (F) converges in distribution to
a tight limit in £00 (F) if and only if t
sup IE*h(G n ) - Eh(G) -+ O.
hEBL] (lOO(F))
We use the notation EM to denote "taking the expectation conditionally on XI, . . . , Xn," or
the expectation with respect to the multinomial vectors M n only. +
t For a metric space IIJ), the set BLI (IIJ)) eonsists of all funetions h : IIJ) r--+ [-1, 1] that are uniformly Lipsehitz:
Ih(ZI) - h(Z2)1 ::s d(ZI, Z2) for every pair (ZI, Z2). See, for example, Chapter 1.12 of [146].
+ For a proof of Theorem 23.7, see the original paper [58], or, for example, Chapter 3.6 of [146].
23.2 Consistency
333
23.7 Theorem (Empirical bootstrap). For every Donsker class F of measurable func-
tions with finite envelope function F,
sup IEMh(CG ) - Eh(CG p ) I O.
hEBLl (lOO(F))
Furthermore, the sequence CG is asymptotically measurable. If P* F 2 < 00, then the
convergence is outer almost surely as well.
Next, consider an analogue of Theorem 23.5, using the functional delta method. Theo-
rem 23.5 goes through without too many changes. However, for many infinite-dimensional
applications of the delta method the condition of continuous differentiability imposed in
Theorem 23.5 fails. This problem may be overcome in several ways. In particular, contin-
uous differentiability is not necessary for the consistency of the bootstrap "in probability"
(rather than "almost surely"). Because this appears to be sufficient for statistical applica-
tions, we shalI limit ourselves to this case.
Consider sequences of maps en and e; with values in a normed space]]) (e.g., £00 (F)) such
that the sequence ,Jn(e n - e) converges unconditionalIy in distribution to a tight random
element T, and the sequence ,Jn (e; - en) converges conditionalI y given XI, X 2, . . . in
distribution to the same random element T. A precise formulation of the second is that
sup IEMh(v'n(e: - en)) - Eh(T)1 o.
h EBL 1 (JI))
Here the notation EM means the conditional expectation given the original data XI, X 2 , . . .
and is motivated by the application to the bootstrap empirical distribution. t By the preceding
theorem, the empirical distribution en == IF n satisfies condition (23.8) if viewed as a map in
£00 (F) for a Donsker class F.
(23.8)
23.9 Theorem (Delta method for bootstrap). Let]]) be a normed space and let ep :]])<t> C
]]) f---+ TRk be Hadamard differentiable at e tangentially to a subspace ]])0. Let en and e; be
map s with values in ]])<t> such that ,Jn(e n - e) T and such that (23.8) holds, in which
,Jn(e; - en) is asymptotically measurable and T is tight and takes its values in ]D)o. Then
the sequence ,Jn (ep (e;) - ep (en)) converges conditionally in distribution to ep (T), given
XI, X 2 , . . . , in probability.
ProoJ. By the Hahn- Banach theorem it is not a loss of generality to assume that the deriva-
tive CjJ : ]D) f---+ TRk is defined and continuous on the whole space. For every h E BL 1 (TRk),
the function h o ep is contained in BLII<t> II OD). Thus (23.8) implies
sup EMh(<P (v'n(e: - en))) - Eh(<p (T)) O.
hEBL 1 (}Rk)
Because Ih (x) - h (y) I is bounded by 2 /\ d (x, y) for every h E BL 1 (TRk),
sup EMh ( v'n(<p(e:) - <p (en) )) - EMh (<p (v'n(e: - en)))
hEBL 1 (}Rk)
< e + 2PM (II v'n( <p (e:) - <p (en)) - <p (v'n(e: - en)) II > e).
(23.10)
t It is assumed that h ( ,jn (e: - en)) is a measurable funetion of M.
334
Bootstrap
The theorem is proved once it has been shown that the conditional probability on the right
converges to zero in outer probability.
The sequence -/ii(e; - en, en - e) converges (unconditionalIy) in distribution to a pair
of two independent copies of T. This folIows, because conditionally given XI, X 2, . . . ,
the second component is deterministic, and the first component converges in distribution
to T, which is the same for every sequence XI, X 2, . . . . Therefore, by the continuous-
mapping theorem both sequences -/ii(e n - e) and -/ii(e; - e) converge (unconditionalIy)
in distribution to separable random elements that concentrate on the linear space lIJ)o. By
Theorem 20.8,
-/ii(ep(e;) - ep(e)) == ep (vn(e; - e)) + 0 (1),
vn (ep (en) - ep (e)) == ep (vn(e n - e)) + o (1).
Subtract the second from the first equation to conclude that the sequence -/ii(ep(e;) -
ep (en)) - ep (-/ii (e; - en)) converges (unconditionalIy) to O in outer probability. Thus,
the conditional probability on the right in (23.10) converges to zero in outer mean. This
concludes the proof. .
23.11 Example (Empirical distributionfunction). Because the cells (-00, tJ c JR form
a Donsker class, the empirical distribution function F n of arandom sample of real-valued
variables satisfies the condition of the preceding theorem. Thus, conditionally on XI, X 2 , . . . ,
the sequence -/ii (ep (F:) - ep (F n) ) converges in distribution to the same limit as -/ii (ep (IF n) -
ep (F)), for every Hadamard-differentiable function ep.
This includes, among others, quantiles and trimmed means, under the same conditions
on the underlying measure F that ensure that empirical quantiles and trimmed means are
asymptoticalIy norma1. See Lemmas 21.3, 22.9, and 22.10. D
23.3 Higher-Order Correctness
The investigation of the performance of a bootstrap confidence interval can be refined by
taking into account the order at which the true level converges to the desired leve1. A
confidence interval is (conservatively) correct at ZeveZ1 - et - tJ up to order O(n- k ) if
P(8n,1 < e < 8 n ,21 p) > 1 - ex - fJ - o ( n 1k ).
Similarly, the quality of the bootstrap estimator for distributions can be assessed more
precisely by the rate at which the Kolmogorov-Smirnov distance between the distribution
function of (en - e) /o-n and the conditional distribution function of (e; - en) /0-; converges
to zero. We shalI see that the percentile t-method usualIy performs better than the per-
centile method. For the percentile t-method, the Kolmogorov-Smimov distance typically
converges to zero at the rate O p (n -1), whereas the percentile method attains "only" an
O p (n- 1 / 2 ) rate of correctness. The latter is comparable to the error of the normal approxi-
mation.
Rates for the Kolmogorov-Smirnov distance translate direct1y into orders of correctness
of one-tailed confidence intervals. The correctness of two-tailed or symmetric confidence
interval s may be higher, because of the cancelIation of the coverage errors contributed by
23.3 Higher-Order Correctness
335
the left and right tails. In many cases the percentile method, the percentile t - methad, and the
normal approximation all yield correct two-tailed confidence interval s up to order O (n -1).
Their relative qualities may be studied by amore refined analysis. This must also take into
account the length of the confidence intervals, for an increase in length of order O p (n -3/2)
may easily reduce the coverage error to the order O (n -k) for any k.
The technical tool to obtain these results are Edgeworth expansions. Edgewarth's clas-
sical expansion is a refinernent of the central limit theorem that shows the magnitude of the
difference between the distribution function of a sample mean and its normal approxima-
tion. Edgeworth expansions have subsequently been obtained for many other statistics as
well.
An Edgeworth expansion for the distribution function of a statistic (en - e) lan is typically
an expansion in increasing powers of 1 I ,Jfi of the form
P ( e n A- e < x I P ) = <I>(x) + Pl(X I P) <jJ(x) + P2(X I P) <jJ(x) +...
,Jfi n
(23.12)
The remainder is of lower arder than the last included term, uniformly in the argument x.
Thus, in the present case the remain der is o (n -1) (or even O (n - 3 /2) ). The functions Pi
are polynomials in x, whose coefficients depend on the underlying distribution, typically
through (asymptotic) moments of the pair (en, an).
23.13 Example (Sample mean). Let X n be the mean of arandom sample of size n, and
let S == n-I :L7=1 (Xi - X n)2 be the (biased) sample variance. If J-t, a 2 , ),. and K are the
mean, variance, skewness and kurtosis of the underlying distribution, then
- 2
( Xn - J-t ) ),.(x - 1)
p r;; < x I p == <p(x) - r;; cjJ(x)
alv n 6 v n
3K(X 3 - 3x) + ),.2(x 5 - l0x 3 + 15x) ( 1 )
- CjJ (x) + O - .
72n n,Jfi
These are the first two term s of the classical expansion of Edgeworth. If the standard
deviation of the observations is unknown, an Edgeworth expansion of the t -statistic is of
more interest. This takes the form (see [72, pp. 71-73])
( Xn - J-t ) )"(2x2 + 1)
p r;; < x I P = <I>(x) + ,Jfi <jJ(x)
Snlvn 6 n
3K(X 3 - 3x) _2),.2(x 5 + 2x 3 - 3x) -9(x 3 + 3x) ( 1 )
+ cjJ(x) + O r;; '
36n nvn
Although the polynomials are different, these expansions are of the same form. Note that
the polynomial appearing in the 1/,Jfi term is even in both cases.
These expansions generally fail if the underlying distribution of the observations is
discrete. Cramer 's condition requires that the modulus of the characteristic function of the
observations be bounded away from unity on closed interval s that do not contain the origin.
This condition is satisfied if the observations possess a density with respect to Lebesgue
measure. Next to Cramer's condition a sufficient number of moments of the observations
must exist. O
336
Bootstrap
23.14 Example (Studentized quantiles). The pth quantile F- 1 (p) of a distribution func-
tion F may be estimated by the empirical pth quantile JF 1 (p). This is the rth order statistic
of the sample for r equal to the largest integer not greater than np. Its mean square error
can be computed as
E(JF;;] (p) - F_](p))2 = rG) 1] (F-](u) - F_](p))2u r -](1- ut- r du.
An empirical estimator O- n for the mean square error of JF 1 (p) is obtained by replacing
F by the empirical distribution function. If the distribution has a differentiable density f,
then
p( JF;;] (p) F-](p) < x I F) = <I>(x) + P] F) 4>(x) + ° ( n;/4 ).
where Pl (x I F) is the polynomial of degree 3 given by (see [72, pp. 318-321])
3 [ fl ]
P] (x I F) 12 ) p(I - p) = -fii x 3 + 2 - IOp - 12p(I - p) j2 (F-] (p)) x 2
3+6
+ -fii x - 8 + 4p - 12(r - np).
This expansion is unusual in two respects. First, the remainder is of the order O (n- 3 / 4 )
rather than of the order O (n -1). Second, the polynomial appearing in the first term is not
even. For this reason several of the conclusions of this section are not valid for sample
quantiles. In particular, the order of correctness of all empirical bootstrap procedures is
O p (n -1/2), not greater. In this case, a "smoothed bootstrap" based on "resampling" from
a density estimator (as in Chapter 24) may be preferable, depending on the underlying
distribution. O
If the distribution function of (en - e) /o-n admits an Edgeworth expansion (23.12), then
it is immediate that the normal approximation is correct up to order O (1/ ,Jii). Evaluation
of the expansion at the normal quantiles Z f3 and Z I-a yields
P(e n - zf3 o-n < e < en - ZI-a ćJ n I P) == 1 - a - f3
Pl (Zf3 I P)cp(Zf3) - Pl (ZI-a I P)CP(ZI-a) O ( )
+ +.
yn n
Thus, the level of the confidence interval [en - zf3 o-n, en - ZI-a ĆJ n ] is 1 - a - f3 up to
order 0(1/ ,Jii). For a two-tailed, symmetric interval, a and f3 are chosen equal. Inserting
Z f3 == Za == - Z I-a in the preceding display, we see that the errors of order 1/ ,Jii resulting
from the left and right tails cancel each other if P 1 is an even function. In this common
situation the order of correctness improves to O (n -1).
It is of theoretical interest that the coverage probability can be corrected up to any order
by making the normal confidence interval slightly wider than first-order asymptotics would
suggest. The interval may be widened by using quantiles Zan with Ci n < a, rather than Za.
In view of the preceding display, for any an,
A A ( 1 )
P (en - Zan ĆJ n < e < en - ZI-an ĆJ n I P) == 1 - 2a n + O n .
23.3 Higher-Order Correctness
337
The O (n-I) term results from the Edgeworth expansion (23.12) and is universal, indepen-
dent of the sequence an. For an == a - M /n and a sufficiently large constant M, the right
side becomes
1 - 2a + 2: + O ( ) > 1 - 2a - O ( n 1k ).
Thus, a slight widening of the normal confidence interval yields asymptotically correct
(conservative) coverage probabilities up to any order O(n- k ). If o-n == Op(n- 1 / 2 ), then the
widened interval is 2(za n - Za) O-n == O p (n- 3 / 2 ) wider than the normal confidence interval.
This difference is small relatively to the absolute length of the interval, which is O p (n -1/2).
Also, the choice of the scale estimator o-n (which depends on en) influences the width of the
interval stronger than repI ac ing a by an .
An Edgeworth expansion usually remain s valid in a conditional sense if a good estimator
p n is substituted for the true underlying distribution P. The bootstrap version of expansion
(23.12) is
P ( e: -:. en < x I P n ) = <I>(x) + Pl (f n) ep(X) + P2(X I P n) ep(X) + . . . .
n n
In this expansion the remainder term is arandom variable, which ought to be of smaller
order in probability than the last term. In the given expansion the remainder ought to be
o p (n -1 ) uniform1y in x. Subtract the bootstrap expansion from the unconditional expansion
(23.12) to obtain that
( en - 8 ) ( e* - en A )
sup P A < X I p - p n A * < x I p n
x an an
< s p PI(X IP) -:J/I(X I Pn) + P2(X I P) P2(X I Pn) ep(x) + Op ( ).
The polynomials Pi typically depend on P in a smooth way, and the difference P n - p is
typically of the order Op(n- 1 / 2 ). Then the Kolmogorov-Smirnov distance between the true
distribution function of (en - 8)/o-n and its percentile t-bootstrap estimator is of the order
Op(n- 1 ).
The analysis of the percentile method starts from an Edgeworth expansion of the di s tri -
bution function of the unstudentized statistic en - 8. This has as leading term the normal
distribution with variance a;, the asymptotic variance of en - 8, rather than the standard
normal distribution. Typically it is of the form
P(e n - e < X I P) = <1>( :J + Jn ql ( :n P )ep( :J
+ q2 ( P ) ep ( ) +....
n an an
The functions qi are polynomials, which are generally different from the polynomials
occurring in the Edgeworth expansion for the studentized statistic. The bootstrap version
of this expansion is
A* A A ( X ) 1 ( X A ) ( X )
P(e n - en < X I P n) = <I> an + ..jii q l an p n ep an
+ q2 ( p n ) ep ( ) + .. . .
n an an
338
Bootstrap
The Kolmogorov-Smirnov distance between the distribution functions on the left in the pre-
ceding displays is of the same order as the difference between the leading term s ef> Cx / (5 n) -
ef> Cx / an) on the right. Because the estimator an is typically not closer than O p (n- 1/2 ) to eJ,
this difference may be expected to be at best of the order O p Cn- 1/2 ). Thus, the percentile
method for estimating a distribution is correct only up to the order O p (n- 1/2 ), whereas the
percentile t-method is seen to be correct up to the order Op(n- 1 ).
One-sided bootstrap percentile t and percentile confidence interval s attain orders of
correctness that are equal to the orders of correctness of the bootstrap estimators of the
distribution functions: O p (n-I) and O p Cn- 1/2 ), respectively. For equal-tailed confidence
interval s both methods typically have coverage error of the order O p Cn -1 ). The dec-
rease in coverage error is due to the cancellation of the errors contributed by the left and
right tails, just as in the case of normal confidence intervals. The proofs of these assertions
are somewhat technical. The coverage probabilities can be expressed in probabilities of the
type
( en - e A )
P an < n,a I p .
Thus we need an Edgeworth expansion of the distribution of Ce n - e)/a n - n,a' or a related
quantity. A technical complication is that the random variables n,a are only implicitly
defined, as the solution of C23.1).
To find the expansions, first evaluate the Edgeworth expansion for Ce: - en) / &: at its
the upper quantile n,a to find that
C23.15)
I-ex = <P( n,a) + Pl( n,a 1;)<P( n,a) + Op( ).
After expanding <1>, Pl and ej; in Taylor series around Za, we can invert this equation to
obtain the C conditional) Cornish-Fisher expansion
A _ Pl CZa I P) ( 1 )
n,a - Za - -Jii + O p - .
n n
In general, Cornish-Fisher expansions are asymptotic expansions of quantile functions,
much in the same spirit as Edgeworth expansions are expansions of distribution functions.
The probability C23 .15) can be rewritten
P ( e n -8 _ Op ( ) < Za _ PICZa I P) P ) .
an n - -Jii
For a rigorous derivation it is necessary to characterize the O p Cn- l ) term. Informally, this
term should only contribute to terms of order O Cn -1) in an Edgeworth expansion. If we
just ignore it, then the probability in the preceding display can be expanded with the help
of C23.12) as
ef> ( _ PICZaIP) ) PI(Za-n-1/2P1CZaIP)IP) ( _ P1CZaIP) ) O( )
Za r;; + r;; \f-' Za r;; + .
yn yn yn n
The linear term of the Taylor expansion of <t> cancels the leading term of the Taylor expansion
of the middle term. Thus the expression in the last display is equal to 1 - cl up to the order
Problems
339
o (n -1 ), whence the coverage error of a percentile t -confidence interval is of the order
O(n-I).
For percentile interval s we proceed in the same manner, this time inverting the Edgeworth
expansion of the unstudentized statistic. The (conditional) Comish- Fisher expansion for
the quantile n,a of e; - en takes the form
€ ,,, =Z,,- qj(z,,1 Pn) +Op ( ) .
an ,Jn n
The coverage probabilities of percentile confidence interval s can be expressed in probab-
ilities of the type
p(e n - e < n,a I P) = P ( {jn -: e < € ,,, I P ) .
an an
Insert the Cornish- Fisher expansion, again neglect the O p (n -1 ) term, and use the Edgeworth
expansion (23.12) to rewrite this as
<1> ( _ q1(Za I P) ) Pl (Za - n- 1 / 2 q1(Za I P) I P) At ( _ Q1(Za I P) ) O( )
Za r;; + r;; "p Za r;; + .
yn yn yn n
Because Pl and Q1 are different, the cancellation that was found for the percentile t-method
does not occur, and this is generally equal to 1 - a up to the order O (n -1/2). Consequently,
asymmetric percentile intervals have coverage error of the order O (n -1/2). On the other
hand, the coverage probability of the symmetric confidence interval [en - n,a, en - n,l-a]
is equal to the expression in the preceding display minus this expression evaluated for 1 - a
instead of a. In the common situation that both polynomials Pl and ql are even, the term s
of order O(n- 1 / 2 ) cancel, and the difference is equal to 1 - 2a up to the order O(n- 1 ).
Then the percentile two-tailed confidence interval has the same order of correctness as the
symmetric normal interval and the percentile t - interval s .
Notes
For a wider scope on the applications of the bootstrap, see the book [44], whose first
author Efron is the inventar of the bootstrap. Hall [72] gives a detailed treatment of higher-
order expansions of a number of bootstrap schemes. For more information conceming
the consistency of the empirical bootstrap, and the consistency of the bootstrap under the
application of the delta method, see Chapter 3.6 and Section 3.9.3 of [146], or the paper by
Gine and Zinn [58].
PROBLEMS
1. Let fr n be a sequence of random distribution functions and F a continuous, tixed-distribution
function. Show that the following statements are equivalent:
'" p
(i) Fn(x) F(x)foreveryx.
(ii) sup x I fr n (x) - F (x ) I O.
340
Bootstrap
2. Compare in a simulation study Efron's percentile method, the normal approximation in combina-
tion with Fisher's transformation, and the percentile method to set a confidenee interval for the
correlation coefficient.
3. Let X (n) be the maximum of a sample of size n from the uniform distribution on [O, 1], and let
X(n) be the maximum of a sample of size n from the empirical distribution IP n of the first sample.
Show that P(X(n) == X (n) I IPn) -* 1 - e- 1 . What does this mean regarding the consistency of the
empirical bootstrap estimator of the distribution of the maximum?
4. Devise a bootstrap scheme for setting confidenee interval s for f3 in the linear regression model
Yi == a + f3Xi + ei. Show consistency.
5. (Parametric bootstrap.) Let en be an estimator based on observations from a parametric model
Pe such that ,Jn(e n - () - hn/,Jn) converges under () + hn/,Jn to a continuous distribution Le
for every converging sequence h n and every (). (This is slightly stronger than regularity as defined
in the chapter on asymptotic efficiency.) Show that the parametric bootstrap is consistent: If e;
is en computed from observations obtained from Pa ' then ,Jn(e; - en) Le conditionally on
n A A
the original observations, in probability. (The conditionallaw of ,Jn((); - en) is Ln,e if Ln,e is
the distribution of ,Jn(e n - ()) under ().)
6. Suppose that ,Jn(e n -()) T and ,Jn(e; -en) Tin probability given the original observations.
Show that ,Jn( ep (e;) - ep (en)) ep (T) in probability for every map ep that is differentiable
at ().
7. Let Un be a U -statistic based on arandom sample XI, . . . , X n with kemel h(x, y) such that
both Eh(X 1, XI) and Eh 2 (X 1, X2) are fini te. Let U; be the same U -statistic based on a sample
xt,..., X from the empirical distribution of XI,..., Xn. Show that ,Jn(U; - Un) converges
conditionally in distribution to the same limit as ,Jn(U n - ()), almost surely.
8. Suppose that ,Jn(e n -()) T and ,Jn(e; -en) Tin probability given the original observations.
Show that, unconditionally, ,Jn(e n - (), e; - en) (S, T) for independent copies S and T of T.
Deduce the unconditionallimit distribution of ,Jn(e; - ()).
24
Nonparametric Density
Estimation
This chapter is an introduction to estimating densities if the underlying
density of a sample of observations is considered completely unknown,
up to existence of derivatives. We derive rates of convergence for the
mean square error of kernel estimators and show that these cannot be
improved. We also consider regularization by monotonicity.
24.1 Introduction
Statistical model s are called parametric models if they are described by a Euclidean param-
eter (in a nice way). For instance, the binomial model is described by asingle parameter
p, and the normal model is given through two unknowns: the mean and the variance of
the observations. In many situations there is insufficient motivation for using a particular
parametric model, such as a normal model. An alternative at the other end of the scale
is a nonparametric model, which leaves the underlying distribution of the observations
essentially free. In this chapter we discuss one example of a problem of nonparametric
estimation: estimating the density of a sample of observations if nothing is known a priori.
From the many methods for this problem, we present two: kern el estimation and monotone
estimation. Notwithstanding its simplicity, this method can be fully asymptotically efficient.
24.2 Kernel Estimators
The most popular nonparametric estimator of a distribution based on a sample of observa-
tions is the empirical distribution, whose properties are discussed at length in Chapter 19.
This is a discrete probability distribution and possesses no density. The most popular method
of nonparametric density estimation, the kernel method, can be viewed as a recipe to "smooth
out" the pointmasses of sizes 1/ n in order to turn the empirical distribution into a contin-
uous distribution.
Let XI, . . . , X n be arandom sample from a density f on the real line. If we would know
that f belongs to the normal family of densities, then the natural estimate of f would be
the normal density with mean X n and variance S , or the function
1 1 -- 2 2
X f---+ e- 2 (x-X/1) /5/1
sn,J2n
341
342
Nonparametric Density Estimation
_ _ _ _ _ _ _ _ _. -' '_'t . _. _ _ _ __
Figure 24.1. The kemel estimator with normal kernel and two observations for three bandwidths:
small (left), intermediate (center) and large (right). The figures show both the contributions of the
two observations separately (dotted lines) and the kemel estimator (solid lines), which is the sum of
the two dotted lines.
In this section we suppose that we have no prior knowledge of the form of f and want to
"let the data speak as much as possible for themselves."
Let K be a probability density with me an O and variance 1, for instance the standard
normal density. A kernel estimator with kernel or window K is defined as
f(x) = t K ( X - Xi ) .
ni=lh h
Here h is apositive number, still to be chosen, called the bandwidth of the estimator. It
turns out that the choice of the kern el K is far less crucial for the quality of j as an estimator
of f than the choice of the bandwidth. To obtain the best convergence rate the requirement
that K > O may have to be dropped.
Akemei estimator is an example of a smoothing method. The construction of a density
estimator can be viewed as a recipe for smoothing out the total mass 1 over the real line.
Given arandom sample of n observations it is reasonable to start with allocating the total
mass in packages of size lin to the observations. Next akemei estimator distributes the
mass that is allocated to Xi smoothly around Xi, not homogenously, but according to the
kern el and bandwidth.
More formally, we can view a kern el estimator as the sum of n small "mountains" given
by the functions
1 ( x - Xi )
xl--+-K .
nh h
Every small mountain is centred around an observation Xi and has area 1 I nunder it, for
any bandwidth h. For a small bandwidth the mountain is very concentrated (a peak), while
for a large bandwidth the mountain is low and fiat. Figure 24.1 shows how the mountains
add up to asingle estimator. If the bandwidth is small, then the mountains remain separated
and the ir sum is peaky. On the other hand, if the bandwidth is large, the n the sum of the
individual mountains is too fiat. Intermediate values of the bandwidth should give the best
results.
Figure 24.2 shows the kemel method in action on a sample from the normal distribution.
The solid and dotted lines are the estimator and the true density, respectively. The three
pictures give the kern el estimates using three different bandwidths - small, in term edi ate,
and large - each time with the standard normal kern el.
, ,
lI)
I o
II I
£. ,
;:>.. .
. Q)
-r;- OCJ
Q) Q)
'"d
,..,....-I ,..,....-I
ro ro
s a
O O
'"d ro
"? 01)
'"d
'''''';
OCJ
ro ::3
S'O v'O £'0 G'O 'O 0'0
OCJ
Q) Q
...c:
c') c.o
S . ....
O '-"
c.t:: ln
C\J ln
'"d
Q)
N ro
'''''';
OCJ
O
Q)
,..,....-I 1::
\...)
C\J '-"
<x:! o C'I
"'ii OCJ 00
£. ro
O
.-.,
-r;- .
OCJ
'-"
Q) 00
'"d \o
Q) O b
OS '''''';
II OCJ
c.8
Q)
OCJ OCJ '"d
"? Q) ...c: Q)
ro ::3
S '"d b
'''''';
S'O v'O £'0 G'O 'O 0'0 ''''''; Q)
OCJ '"d ...c:
Q)
c') ,..,....-I ro OCJ
Q) "D Q)
a ;>
'''''';
01)
Q) Q)
C\J
N .,....;
,..,....-I
'''''';
'"d '"d
Q)
Q)
Q)
Qj O
...c: '"d
= Q)
...c:
.- c.8
co
«)
ci o
II
£.
-r;-
24.2 Kernel Estimators
S'O
£'0
'O
G'O
V'O
343
I
I c')
I
I
,
,
C\J
"?
0'0
344
N onparametric Density Estimation
A popular criterion to judge the quality of density estimators is the mean integrated
square error (MISE), which is defined as
MISEfeJ) = f Ef (f(x) - f(x))2 dx
= f varff(x)dx + f (Eff(x) - f(x)f dx.
This is the mean square error Ef (! (x) - f (x) ) 2 of ! (x) as an estimator of f (x) integrated
over the argument x. If the mean integrated square error is small, then the function f is
close to the function f. (We assume that! n is jointly measurable to make the mean square
error well defined.)
As can be seen from the second representation, the mean integrated square error is the
sum of an integrated "variance term" and a "bias term." The mean integrated square error
can be small only if both terms are small. We shalI show that the two terms are of the orders
1
and h 4 ,
nh'
respectively. Then it follows that the variance and the bias terms are balanced for (nh) -1 r-v
h 4 , which implies an optimal choi ce of bandwidth equal to h r-v n -1/5 and yields a mean
integrated square error of order n -4/5.
Informally, these orders follow from simple Taylor expansions. For instance, the bi as of
f(x) can be written
f 1 ( x-t )
Ef f(x) - f(x) = h K h f(t) dt - f(x)
= f K(y)(f(x-hY)-f(x))dy.
Developing f in a Taylor series around x and using that f y K (y) dy == O, we see, informally,
that this is equal to
f lK(y)dy!h 2 j"(x)+....
Thus, the squared bi as is of the order h 4 . The variance term can be handled similarly. A
precise theorem is as follows.
24.1 Theorem. Suppose that f is twice continuously differentiable with f I f" (x) 1 2 dx <
00. Furthermore, suppose that f y K (y) d y == O and that both f y2 K (y) dy and f K 2 (y) dy
are finite. Then there exists a constant C f such that for small h > O
f Ef (f(x) - f(x))2 dx < Cf C + h 4 ).
Consequently, for h n r-v n- 1 / 5 , we have MISEf(!n) == O(n- 4 / 5 ).
,
24.2 Kernel Estimatars
345
Proof. Because a kernel estimator is an average of n independent random variables, the
variance of f (x) is (1/ n) times the variance of one term. Hence
varff(x) = varf K( x XI ) < n 2 EfK2( X XI )
== / K2(Y)f(X - hy) dy.
nh
Take the integral with repect to x on both left and right sides. Because J f (x - hy) dx == 1
is the same for every value of hy, the right side reduces to (nh)-l J K 2 (y) dy, by Fubini's
theorem. This concludes the pro of for the variance term.
To upper bound the bias term we first write the bias Ef f (x) - f (x) in the form as given
preceding the statement of the theorem. N ext we insert the formula
j(x + h) - j(x) = hj'(x) + h211 f"(x + sh)(l - s) ds.
This is a Taylor expansion with the Laplacian representation of the remainder. We obtain
Ef f(x) - j(x) = III K (y) [-hy J' (x) + (hy)2 j" (x - shy)(1 - s)] ds dy.
Because the kernel K has mean zero by assumption, the first term inside the square brackets
can be deleted. U sing the Cauchy-Schwarz inequality (EU V)2 < EU 2 E V 2 on the variables
U == Y and V == Yf"(x - ShY)(l - S) for Y distributed with density K and S uniformly
distributed on [O, 1] independent of Y, we see that the square of the bias is bounded above by
h 4 I K(y)l dy III K(y)l j"(x - shy)2 (1 - s)2 ds dy.
The integral of this with respect to x is bounded above by
h 4 (I K(y)y2 dy r I j"(x)2 dx .
This concludes the derivati on for the bias term.
The last assertion of the theorem is trivial. .
The rate O(n- 4/5 ) for the mean integrated square eITor is not impressive if we compare
it to the rate that could be achieved if we knew a priori that f belonged to some parametric
family of densities fe. Then, likely, we would be able to estimate e by an estimator such
that e == e + O p (n- 1/2 ), and we would expect
MISE e (Je) = I Ee (Je(x) - je(x))2 dx Ee(8 - e)2 = o ( ).
This is a factor n -1 IS smaller than the mean square error of a kernel estimator.
This loss in efficiency is onlyamodest price. After all, the kern el estimator works for
every density that is twice continuously differentiable whereas the parametric estimator
presumably fails miserably if the true density does not belong to the postulated parametric
model.
346
Nonparametric Density Estimation
Moreover, the lost factor n- 1 / 5 can be (almost) covered if we assume that f has suffi-
ciently many derivatives. Suppose that f is m times continuously differentiable. Drop the
condition that the kemel K is a probability density, but use akemei K such that
!K(Y)d Y =l, !YK(Y)dY=O,..., !y m - 1 K(Y)d Y =O,
!IYlmK(Y)d Y < 00, ! K\y)dy < 00.
Then, by the same arguments as before, the bias term can be expanded in the fonn
Ej f(x) - f(x) = ! K(y)(J(x - hy) - f(x)) dy
== ! K(Y) (_l)mhmym f(m)(x) dy +. ..
ml
Thus the squared bi as is of the order h 2m and the bias-variance trade-off (nh)-1 rv h 2m
is solved for hrv n 1 /(2m+1). This leads to a mean square error of the order n- 2m /(2m+1),
which approaches the "parametric rate" n-I as m 00. This claim is made precise in the
following theorem, whose proof proceeds as before.
24.2 Theorem. Suppose that f is m times continuously differentiable with JI j(m) (x) 1 2
dx < 00. Then there exists a constant C f such that for small h > O
! Ej (f(x) - f(x))2 dx < ej ( n 1 h + h2m).
Consequently, for h n rv n- 1 /(2m+1), we have MISEf(Jn) == o (n- 2m /(2m+I)).
In practice, the number of derivatives of f is usually unknown. In order to choose a
proper bandwidth, we can use cross- validation procedures. These yield a data-dependent
bandwidth and also solve the problem of choosing the constant preceding h- I /(2m+I). The
combined procedure of density estimator and bandwidth selection is called rate-adaptive
if the procedure attains the upper bound n- 2m /(2m+I) for the mean integrated square error
for every m.
24.3 Rate Optimality
In this section we show that the rate n- 2m /(2m+I) of akemei estimator, obtained in The-
orem 24.2, is the best possible. More precisely, we prove the following. Inspection of
the proof of Theorem 24.2 reveals that the constants C f in the upper bound are uniformly
bounded in f such that JI f(m) (x) 1 2 dx is uniformly bounded. Thus, letting :0n,M be the
class of all probability densities such that this quantity is bounded by M, there is a constant
Cm,M such that the kern el estimator with bandwidth h n == n- I /(2m+I) satisfies
! ( 1 ) 2m/(2m+l)
sup Ef (Jn(x) - f(x))2 dx < Cm,M - .
f E:F m . M n
24.3 Rate Optimality
347
In this section we show that this upper bound is sharp, and the kernel estimator rate optimal,
in that the maximum risk on the left side is bounded below by a similar expression for every
density estimator i n' for every fixed m and M.
The proof is based on a construction of subsets Fn c Fm,M, consisting of 2 rn functions,
with r n == Ln 1j (2m+1) J, and on bounding the supremum over Fm,M by the average over Fn.
Thus the number of elements in the average grows fairly rapidly with n. An approach,
such as in section 14.5, based on the comparison of in at only two elements of Fm,M does
not seem to work for the integrated risk, although such an approach readily yields a lower
bound for the maximum risk sup f Ef (in (x) - f (x) ) 2 at a fixed x.
The subset :F n is indexed by the set of all vectors e E {O, 1 }r n consisting of sequences
of r n zeros or ones. For h n == n -lj(2m+ 1), let X n ,l < X n ,2 < ... < Xn,n be a regular grid
of meshwidth 2h n . For a fixed probability density f and a fixed function K with support
(-1, 1), define, for every e E {O, 1 }r n ,
r n ( )
m X - X n j
fn,B(X) == f(x) + h n L ejK ' .
. 1 hn
J=
If f is bounded away from zero on an interval containing the grid, I K I is bounded, and
f K (x) dx == O, then fn,e is a probability density, at least for large n. Furthermore,
/lfn(;>C X ) 1 2 dx < 2 /lf(m)(x)1 2 dx + 2h n r n /IK(m\x)1 2 dx.
It follows that there exist many choices of f and K such that fn,B E :Fm,M for every e.
The following lemma gives a lower bound for the maximum risk over the parameter set
{O, l}r, in an abstract form, applicable to the problem of estimating an arbitrary quantity
1/1(e) belonging to a metric space (with metric d). Let H(e, ef) == L =l lei - eri be the
Hamming distance on {O, l}r, which counts the number of positions at which e and ef
differ. For two probability measures P and Q with densities p and q, write II P /\ Q II for
fp/\qd .
24.3 Lemma (Assouad). For any estimator T based on an observation in the experiment
( p B : e E {O, I} r ), and any p > O,
d P (1/1(e), 1/1 (ef) ) r
max 2 P E B d P (T, 1/1 (e)) > min - min II P B /\ PB'II.
B H (B ,B')?:.l H (e, ef) 2 H (B ,B')= 1
Proof. Define an estimator S, taking its values in e == {O, l}r, by letting S == e if ef r-+
d (T, 1/1 (ef)) is minimalover e at ef == e. (If the minimum is not unique, choose a point of
minimum in any consistent way.) By the triangle inequality, for any e, d ( 1/1 (S), 1/1 (e)) <
d(1/1(S), T) + d(1/1(e), T), which is bounded by 2d(1/1(e), T), by the definition of S. If
d P (1/1 (e), 1/1 (ef)) > aH (e, ef) for all pairs e # ef, then
2 P E B d P (T, 1/1 (e)) > EBd P (1/1 (S), 1/1 (e)) > aE B H (S, e).
The maximum of this expression over e is bounded below by the average, which, apart
348
Nonparametric Density Estimation
from the factor a, can be written
1 r 1 r ( 1 f 1 f )
-""EBIS.-e'I==-" -" S.dP B +- (l-S.)dPB.
2r /-; J J 2 2r-I J 2 r -I J
B }=I }=I B: Bj=O B : Bj=I
This is minimized over S by choosing S) for each j separately to minimize the jth term in
the sume The expression within brackets is the sum of the error probabilities of a test of
- 1 L
p .-- p
O,J - 2 r -I B,
B : B j =0
- 1 L
versus Pl J . == _ 1 P B .
, 2 r -
B : Bj=I
Equivalently, it is equal to 1 minus the difference of power and level. In Lemma 14.30 this
wasseentobeatleast 1- II P o,} - P I,}II == II P O,} /\ P I,}II. Hencetheprecedingdisplay
is bounded below by
1 - -
2 II PO,} /\ Pl,} II.
}=I
Because the minimum p m /\ q m of two averages of numbers is bounded below by the average
m- I L Pi /\ qi of the minima, the same is true for the total variation norm of a minimum:
II P m /\ Qm ll > m- I L II Pi /\ Qill. The2 r - I terms P B and PB' in theaverages P O,} and P l,}
can be ordered and matched such that each pair e and ef differ only in their jth coordinate.
Conclude that the preceding display is bounded below by LJ=I min II PB /\ PB'II, in which
the minimum is taken over all pairs e and ef that differ by exactly one coordinate. .
We wish to apply Assouad's lemma to the product measures resulting from the densities
In,B. Then the following inequality, obtained in the proof of Lemma 14.31, is useful. It
relates the total variation, affinity, and Hellinger distance of product measures:
Ilp n /\ Qnll > A2(pn, Qn) == (1 _ H2(p, Q))2n.
24.4 Theorem. There exists a constant Dm,M such that for any density estimator f n
f ( 1 ) 2ml(2m+I)
sup Ef (fn(x) - f(x))2 dx > Dm,M n .
fEFm,M
Proof. Because the functions In,B are bounded away from zero and infinity, uniformly in
e, the squared Hellinger distance
f( i I/2 - i I/ : ) 2 dx == f ( fn,B - fn,e' ) 2 dx
n,e n,B .{I/2 .{I/2
Jn,B + Jn,e'
is up to constants equal to the squared L 2 -distance between In,e and In,BI. Because the
24.4 Estimating a Unimodal Density
349
functions K ((x - Xn,j ) / h n ) have disjoint supports, the latter is equal to
h m le j - ej 1 2 f K 2 ( x :n,j ) dx = h m+l H(e, el) f K 2 (x) dx.
This is of the order 1/ n. Inserting this in the lower bound given by Assouad' s lemma, with
1/1 (e) == In,e and d (1/1 (e), 1/1 (el)) the L 2 -distance, we find up to a constant the lower bound
h m + 1 (r n /2) (1 - O ( 1 / n) ) 2n . .
24.4 Estimating a Unimodal Density
In the preceding sections the analysis of nonparametric density estimators is based on the
assumption that the true density is smooth. This is appropriate for kemel-density estimation,
because this is a smoothing method. It is also sensible to place some a priori restriction on
the true density, because otherwise we cannot hope to achieve much beyond consistency.
However, smoothness is not the only possible restriction. In this section we assume that the
true density is monotone, or unimodal. We start with mon oto ne densities and next view a
unimodal density as a combination of two monotone pieces.
It is interesting that with monotone densities we can use maximum likelihood as the
estimating principle. Suppose that XI, . . . , X n is arandom sample from a Lebesgue density
I on [O, (0) that is known to be nonincreasing. Then the maximum likelihood estimator
f n is defined as the nonincreasing probability density that maximizes the likelihood
n
I r-+ OI(X i ).
i=1
This optimization problem would not have a solution if I were only restricted by possessing
a certain number of derivatives, because very high peaks at the observations would yield an
arbitrarily large likelihood. However, under monotonicity there is a unique solution.
The solution must necessarily be a left-continuous step function, with steps only at the
observations. Indeed, if for a given I the limit from the right at X(i-1) is bigger than the
limit from the left at X (i), then we can redistribute the mass on the interval (X (i -1), X (i)]
by raising the value I (X (i)) and lowering I (X (i -1) + ), for instance by setting I equal to
the constant value (X(i) - X(i_l))-1 f x X i) I(t) dt on the whole interval, resulting in an
(1-1)
increase of the likelihood. By the same reasoning we see that the maximum likelihood
estimator must be zero on (X (n), (0) (and (-00, O)). Thus, with li == f n (X (i)), finding the
maximum likelihood estimator reduces to maximizing [17=1 li under the side conditions
(with X (O) == O)
11 > 12 > . . . > In > O,
n
Lli (X(i) - X(i-1)) == 1.
i=1
This problem has a nice graphical solution. The least concave majorant of the empirical
distribution function IF n is defined as the smallest concave function ft n with ft n (x) > IF n (x)
for every x. This can be found by attaching a rope at the origin (O, O) and winding this
(from above) around the empirical distribution function IF n (Figure 24.3). Because ft n is
350
Nonparametric Density Estimation
o
co
o
(o
o
o
C\J
o
o
o
o
2
3
Figure 24.3. The empirical distribution and its concave majorant of a sample of size 75 from the
exponential distribution.
LO
o
o
o
o
2
3
Figure 24.4. The derivative of the concave majorant of the empirical distribution and the true density
of a sample of size 75 from the exponential distribution.
concave, its derivative is nonincreasing. Figure 24.4 shows the derivative of the concave
majorant in Figure 24.3.
24.5 Lemma. The maximum likelihood estimator f n is the left derivative of the least
concave majorant F n of the empirical distribution JF n, that is, on each of the intervals
(X U - 1 )' XCi)J it is equal to the slope of Fn on this interval.
Proof. In this proof, let f n denote the left derivative of the least concave majorant. We
shaH show that this maximizes the likelihood. Because the maximum likelihood estimator
24.4 Estimating a Unimodal Density
351
is necessarily constant on each interval (X U - 1 )' X(i)], we may restrict ourselves to densities
f with this property. For such an f we can write log f == L ai 1 [O,X(i)] for the constants
ai == log Ji / fi+l (with fn+l == 1), and we obtain
f log f dFn = tai Fn(X(i)) > tai JFn(X(i)) = f log f dJF n .
i=1 i=1
For f == in this becomes an equality. To see this, let YI < Y2 < ... be the points where
F n touches lF n. Then i n is constant on each of the intervals (Yi -1, Yi], so that we can write
log J n == L bi 1 [O,Yi]' and obtain
f logfn dFn = LbiFn(Yi) = Lb;JFn(y;) = f log fn dJF n .
Third, by the identifiability property of the Kullback-Leibler divergence (see Lemma 5.35),
for any probability density f,
f log f n d F n > f log f d F n,
with strict inequality unless i n == f. Combining the three displays, we see that J n is the
unique maximizer of f r--+ f log f dJF n. .
Maximizing the likelihood is an important motivation for taking the derivative of the
concave majorant, but this operation also has independent value. Taking the concave majo-
rant (or convex minorant) of the primitive function of an estimator and next differentiating
the result may be viewed as a "smoothing" device, which is useful if the target function
is known to be monotone. The estimator i n can be viewed as the result of this procedure
applied to the "naive" density estimator
- 1
f n (x) = ( ) ,
n X(i) - X U - 1 )
This function is very rough and certainly not suitable as an estimator. Its primitive function
is the polygon that linearly interpolates the extreme points of the empirical distribution
function JF n, and its smallest concave majorant coincides with the one of JF n. Thus the
derivative of the concave majorant of F n is exactly in.
Consider the rate of convergence of the maximum likelihood estimator. Is the assumption
of monotonicity sufficient to obtain a reasonable performance? The answer is affirmative
if a rate of convergence of n 1/3 is considered reasonable. This rate is slower than the rate
n m/ (2m+ 1) of a kern el estimator if m > 1 derivatives exist and is comparable to this rate given
one bounded derivative (even though we have not established a rate under m == 1). The rate
of convergence n 1/3 can be shown to be best possible if only monotonicity is assumed. It
is achieved by the maximum likelihood estimator.
X E (XU-l)' X(i)J.
24.6 Theorem. If the observations are sampled from a compactly supported, bounded,
monotone density f, then
f (fn(x) - f(x))2 dx = Op(n- 2/3 ).
352
Nonparametric Density Estimation
Proof. This result is a consequence of a general result on maximum likelihood estimators
of densities (e.g., Theorem 3.4.4 in [146].) We shall give amore direct proof using the
convexity of the class of monotone densities.
The sequence II/n 1100 == / n (O) is bounded in probability. Indeed, by the characterization
of / n as the slope of the concave majorant of JF n, we see that / n (O) > M if and only if there
exists t > O such that JF n (t) > Mt. The claim follows, because, by concavity, F (t) < I (O)t
for every t, and, by Daniel's theorem ([134, p. 642]),
( JFn(t) ) 1
p sup > M == -.
t>O F(t) M
It follows that the rate of convergence of / n is the same as the rate of the maximum likelihood
estimator under the restriction that I is bounded by a (1arge) constant. In the remainder of
the proof, we redefine / n by the latter estimator.
Denote the true density by lo. By the definition of /n and the inequality logx < 2( -
1 ),
o < JF n log 1 A in 1 < 2JF n ( A 2in 1 ) .
"2 I n + "2 lo I n + lo -
Therefore, we can ob tain the rate o f convergence of j n by an application of Theorem 5.52
or 5.55 with m f == v' 21/(1 + lo).
Because (m f - m fo)(/o - I) < O for every I and lo it follows that Fo(m f - m fo) <
F(m f - m fo) and hence
Fo(mf - mfo) < (Fo + F)(mf - mfo) == _ h2(/, I + /o);S - h 2 (/, lo),
in which the last inequality is elementary calculus. Thus the first condition of Theorem 5.52
is satisfied relative to the Hellinger distance h, with a == 2.
The map I 1---+ m f is increasing. Therefore, it turns brackets [11, 12] for the functions
x 1---+ I(x) into brackets [m fl' m 12] for the functions x 1---+ m f(x). The squared L 2 (F o )-size
of these brackets satisfies
Fo(m fl - m f2)2 < 4h 2 (/1' 12).
It follows that the L 2 (Fo)-bracketing numbers of the class of functions m f can be bounded
by the h-bracketing numbers of the functions I. The latter are the L 2 (A)-bracketing numbers
of the functions y'J, which are monotone and bounded by assumption. In view of Example
19.11,
1
logN[](2s, {mf: I EF}, L 2 (F o )) < 10gN[](s, H, L 2 (A));S-.
s
Because the functions m f are uniformly bounded, the maxiInal inequality Lemma 19.36
gives, with J (8) == f VTTi ds == 2 ,
Efa sup I CGn(/-lo)I;S ( l+ 0n ) .
h(f,fo)<8 8 n
Therefore, Theorem 5.55 applies with CPn (8) equal to the right side, and the Hellinger
distance, and we conclude that he/n' lo) == Op(n- 1 / 3 ).
24.4 Estimating a Unimodal Density
353
I
I
I
I
I
I
I
I
1\ -1
f n (a) x
Figure 24.5. If In (x) :::: a, then a line of slope a moved down vertically from +00 first hits IFn to
the left of x. The point where the line hit s is the point at which IF n is farthest above the line of slope
a through the origin.
The L 2 (A)-distance between uniformly bounded densities is bounded up to a constant
by the Hellinger distance, and the theorem follows. .
The most striking known results about estimating amonotone density concern limit
distributions of the maximum likelihood estimator, for instance at a point.
24.7 Theorem. If f is differentiable at x > O with derivative f' (x) <O, then, with {Z(h) :
h E JR} a standard Brownian motion process (two-sided with Z(O) = O),
n 1 / 3 (Jn(x) - f(x)) 14f'(X)f(x)11/3 argmax{Z(h) - h2}.
hEIR
ProoJ. For simplicity we assume that f is continuously differentiable at x. Detine a
stochasticprocess {J 1(a):a > O} by
J 1 (a) = argmax {IF n (S) - as } ,
S O
in which the largest value is chosen when multiple maximizers exist. The suggestive notation
is justified, as the function J 1 is the inverse of the maximum likelihood estimator J n in that
!n(x) < a if and only if j 1(a) < x, for every x and a. This is explained in Figure 24.5.
We first derive the limit distribution of ! 1 . Let On = n -1/3.
By the change of variable s x + hOn in the detinition of J 1, we have
n1/3(J 1 o f(x) - x) = argmax{IFn(x + hOn) - f(x)(x + ho n )}.
h -nl/3 x
Because the location of a rnaximum does not change by a vertical shift of the whole function,
we can drop the term f(x)x in the right side, and we may add a term IFn(x). For the same
354
Nonparametric Density Estimation
reason we may also multiply the process in the right side by n 2 / 3 . Thus the preceding
display is equal to the point of maximum h n of the process
n 2/3 [ OF n - F) (x + h 8n) - (lF n - F) (x ) ] + n 2/3 [ F (x + h 8n) - F (x) - f (x ) h 8n ] .
The first term is the local empiric al pro cess studied in Example 19.29, and converges in
distribution to the process h 1---+ -J f(x) Z(h), for Z a standard Brownian motion process,
in fOO(K), for every compact interval K. The second term is a deterministic "drift" process
and converges on compacta to h 1---+ ! f' (x )h 2 . This suggests that
nl/3(f 1 o f(x) - x) argmax{ yI f(x) Z(h) + f'(X)h2}.
hEJR
This argument remains valid if we replace x by x n == x - 8nb throughout, where the limit
is the same for every b E IR.
We can write the limit in amore attractive form by using the fact that the processes h 1---+
Z((5h) and h 1---+ -JCjZ(h) are equal in distribution for every (5 > O. First, apply the ch ange
of variables h 1---+ (5 h, next pull (5 out of Z( (5 h), then divide the pr ocess b y -J f (x)(5 , and
finally choose (5 such that the quadratic termreduces to -h 2 , that is -J f (x)a == -! f' (x )a 2 .
Then we obtain, for every b E IR,
( ) 2/3
n 1/3 (f 1 o f(x - 8n b ) - (x - 8n b )) f X) argmax {Z(h) - h2}.
-"2f (x) hEJR
The connection with the limit distribution of f n (x) is that
P(n 1 / 3 (fn(x) - f(x)) < -bf'(x)) = P(fn(x) < f(x - 8n b ) + 0(1))
= P(nl/3(f 1 o f(x - 8nb) - (x - 8nb)) < b) + 0(1).
Combined with the preceding display and simple algebra, this yields the theorem.
The preceding argument can be made rigorous by application of the argmax continuous-
mapping theorem, Corollary 5.58. The limiting Brownian motion has continuous sample
paths, and maxima of Gaussian processes are automatically unique (see, e.g., Lemma 2.6 in
[87]). Therefore, we ne ed only check that h n == Op(l), for which we apply Theorem 5.52
with
mg == l[o,x n +gJ - l[o,x n J - f(xn)g.
(In Theorem 5.52 the function mg can be allowed to depend on n, as is clear from its
generalization, Theorem 5.55.) By its definition, gn == 8nhn maximizes g 1---+ JF nmg, whence
we wish to show that gn == O p (8n). By Example 19.6 the bracketing numbers of the class
of functions {1 [O,xn+gJ - 1 [O,xnJ : 19 I < 8} are of the o rder 8/£2; the envelope function
I 1 [O,x n +8J - l[o,x n JI has L 2 (F)-norm of the order -J f(x)8. By Corollary 19.35,
l V1> g
E sup IGnmgl log 2" d£ 05.
Igl<8 o £
By the concavity of F, the function g 1---+ F (x n + g) - F (x n ) - f (x n ) g is nonpositive
and nonincreasing as g moves away from O in either direction (draw a picture.) Because
24.4 Estimating a Unimodal Density
355
f' (xn) -+ f' (x) < O, there exists a constant C such that, for sufficiently large n,
Fmg == F(x n + g) - F(x n ) - f(xn)g < -C(g2 1\ Igl).
If we would know already that gn O, then Theorem 5.52, applied with a == 2 and fJ == ,
yields that gn == O p (8n).
The consistency of gn can be shown by a direct argument. By the Glivenko-Cantelli
theorem, for every 8 > O,
sup Fnmg < sup Fmg + opel) < -C inf (g2 1\ 19 I) + opel).
Igl::::8 Igl::::8 Igl::::8
Because the right side is strictly smaller than O == IF nmO, the maximizer gn must be contained
in [-8, 8] eventually. .
Results on density estimators at a point are perhaps not of greatest interest, because it is
the overall shape of a density that counts. Renee it is interesting that the preceding theorem
is also true in an LI -sense, in that
n 1 / 3 f lln(X) - f(x)1 dx f I4 fl (X)f(x)1 1 / 3 dx Eargmax {Z(h) - h2}.
hEJR
This is true for every strictly decreasing, compactly supported, twice continuously differ-
entiable true density f. For boundary cases, such as the uniform distribution, the behavior
of I n is very different. N ote that the right side of the preceding display is degenerate. This
is explained by the fact that the random variables n 1 / 3 (ln(x) - f(x)) for different values
of x are asymptotically independent, because they are only dependent on the observations
Xi very close to x, so that the integral aggregates a large number of approximately indepen-
dent variables. It is also known that n 1/6 times the difference between the left side and the
right sides converges in distribution to a normal distribution with mean zero and variance
not depending on f. For uniformly distributed observations, the estimator In (x) remains
dependent on all n observations, even asymptotically, and attains a ,Jn-rate of convergence
(see [62]).
We define a density f on the real line to be unimodal if there exists a number M f, such
that f is nondecreasing on the interval (- 00, M f] and nondecreasing on [M f, (0). The
mode M f need not be unique. Suppose that we observe arandom sample from a unimodal
density.
If the true mode M f is known a priori, then a natural extension of the preceding discussion
is to estimate the distribution function F of the observations by the distribution function
ft n that is the least concave majorant of IFn on the interval [M f, (0) and the greatest convex
minorant on (-00, M f]. Next we estimate f by the derivative In of ft n. Provided that
none of the observations takes the value M f, this estimator maximizes the likelihood, as
can be shown by arguments as before. The limit results on monotone densities can also be
extended to the present case. In particular, because the key in the proof of Theorem 24.7
is the characterization of I n as the derivative of the concave majorant of IF n, this theorem
remains true in the unimodal case, with the same limit distribution.
If the mode is not known a priori, then the maximum likelihood estimator does not exist:
The likelihood can be maximized to infinity by placing an arbitrary large mode at some
fixed observation. It has been proposed to remedy this problem by restricting the likelihood
356
Nonparametric Density Estimation
to densities that have a modal interval of a given length (in which f must be constant and
maximal). Altematively, we could estimate the mode by an independent method and next
apply the procedure for a known mode. Both of these possibilities break down unless f
possesses some additional properties. A third possibility is to try every possible value M as
amode, calculate the estimator iM for known mode M, and select the best fitting one. Bere
"best" could be operationalized as (nearly) minimizing the Kolmogorov-Smirnov distance
IIFnM -1F n 1100' It can be shown (see [13]) that this procedure renders the effect of the mode
being unknown asymptotically negligible, in that
!IJ:(x) - J f(x)ldx < 41IJFn - Flloo = Op( Jn ).
up to an arbitrarily small tolerance parameter if NI only approximately achieves the mini-
mumof M II Fn M -1F n 1100' This extra "error" is oflowerorderthan the rate of convergence
n 1/3 of the estimator with a known mode.
Notes
The literature on nonparametric density estimation, or "smoothing," is large, and there is an
equally large literature concerning the parallei problem of nonparametric regression. N ext
to kernel estimation popular methods are based on classical series approximations, spline
functions, and, most recent1y, wavelet approximation. Besides different rnethods, a good
deal is known concerning other loss functions, for instance L 1 -loss and automatic rnethods
to choose a bandwidth. Most recent1y, there is a revived interest in obtaining exact constants
in mini max bounds, rather thanjust rates of convergence. See, for instance, [14], [15], [36],
[121], [135], [137], and [148] for introductions and further references. The kernel estimator
is often named after its pioneers in the 1960s, Parzen and Rosenblatt, and was originally
developed for smoothing the periodogram in spectral density estimation.
A lower bound for the maximum risk over Holder classes for estimating a density at a
single point was obtained in [46]. The lower bound for the L2-risk is more recent. Birge
[12] gives a systematic study of upper and lower bounds and their relationship to the rnetric
entropy of the model. An alternative for Assouad's lemma is Fano's lemma, which uses the
Kullback - Leibler distance and can be found in, for example, [80].
The maximum likelihood estimator for amonotone density is often called the Grenander
estimator, after the author who first characterized it in 1956. The very short proof of
Lemma 24.5 is taken from [64]. The limit distribution of the Grenander estimator at a
point was first obtained by Prakasa Rao in 1969 see [121]. Groeneboom [63] gives a
characterization of the limit distribution and other interesting related results.
PROBLEMS
1. Show, informally, that under sufficient regularity conditions
MISEf(f) n j K2(Y)dY+ h4j jlf(X)2 d x (j y 2 K(Y)d Y Y
What does this imply for an optimal choice of the bandwidth?
Problems
357
2. Let XI, . . . , X n be arandom sample from the normal distribution with variance 1. Calculate the
mean square error of the estimator ep (x - X n) of the common density.
3. Using the argument of section 14.5 and a submodel as in section 24.3, but with r n == 1, show that
the best rate for estimating a density at a fixed point is also n- m /(2m+l).
4. Using the argument of section 14.5, show that the rate of convergence n 1 / 3 of the maximum
likelihood estimator for amonotone density is best possible.
5. (Marshall's lemma.) Suppose that F is concave on [O, (0) with F (O) == O. Show that the least
concave majorant Fn of 1F n satisfies the inequality II F n - F 1100 :s 111F n - F 1100' What doe s this
imply about the limiting behavior of F n ?
25
Semiparametric Models
This chapter is concerned with statistical models that are indexed by
infinite-dimensional parameters. It gives an introduction to the theory
of asymptotic efficiency, and discusses methods of estimation and
testing.
25.1 Introduction
Semiparametric models are statistical model s in which the parameter is not a Euclidean
vec tor but ranges over an "infinite-dimensional" parameter set. A different name is "model
with a large parameter space." In the situation in which the observations consist of a ran-
dom sample from a common distribution P, the model is simply the set P of all possible
values of P: a collection of probability measures on the sample space. The simplest type
of infinite-dimensional model is the nonparametric model, in which we observe arandom
sample from a completely unknown distribution. Then P is the colIection of all probability
measures on the sample space, and, as we shalI see and as is intuitively clear, the empirical
distribution is an asymptotically efficient estimator for the underlying distribution. More
interesting are the intermediate models, which are not "nicely" parametrized by a Euclidean
parameter, as are the standard classical models, but do restrict the distribution in an im-
portant way. Such model s are often parametrized by infinite-dimensional parameters, such
as distribution functions or densities, that express the structure under study. Many aspects
of these parameters are estimable by the same order of accuracy as classical parameters,
and efficient estimators are asymptoticalIy normal. In particular, the model may have a
natural parametrization ((), 1]) 1---+ P e , rJ' where () is a Euclidean parameter and 17 runs through
a nonparametric class of distributions, or some other infinite-dimensional set. This gives a
semiparametric model in the strict sense, in which we aim at estimating () and consider 17
as anuisanee parameter. More generallY' we focus on estimating the value 1jf(P) of some
function vr : P 1---+ IR. k on the model.
In this chapter we extend the theory of asymptotic efficiency, as developed in Chapters 8
and 15, from parametric to semiparametric model s and discuss some methods of estimation
and testing. Although the efficiency theory (lower bounds) is fairly complete, there are stiH
important holes in the estimation theory. In particular, the extent to which the lower bounds
are sharp is unclear. We limit ourselves to parameters that are ,Jn-estimable, although
in most semiparametric models there are many "irregular" parameters of interest that are
outside the scope of "asymptoticaHy normai" theory. Semiparametric testing theory has
358
25.1 Introduction
359
little more to ofter than the comforting conclusion that tests based on efficient estimators
are efficient. Thus, we shalI be brief about it.
We conclude this introduction with a list of examples that shows the scope of semipara-
metric theory. In this description, X denotes a typical observation. Random vectors Y, Z,
e, and f are used to describe the model but are not necessarily observed. The parameters
e and v are always Euclidean.
25.1 Example (Regression). Let Z and e be independent random vectors and suppose
that Y == ILe (Z) + ae (Z)e for functions ILe and ae that are known up to e. The observation
is the pair X == (Y, Z). If the distribution of e is known to belong to a certain paramet-
ric family, such as the family of N (O, a 2 )-distributions, and the independent variables Z
are modeled as constants, then this is just a classical regression model, allowing for het-
eroscedasticity. Semiparametric versions are obtained by letting the distribution of e range
over all distributions on the real line with mean zero, or, alternatively, over all distributions
that are symmetric about zero. D
25.2 Example (Projection pursuit regression). Let Z and e be independent random vec-
tor s and let Y == rJ(e T Z) + e for a function rJ ranging over a set of (smooth) functions, and e
having an N (O, a 2 )-distribution. In this model e and rJ are confounded, but the direction of
e is estimable up to its sign. This type of regres sion model is also known as a single-index
model and is intermediate between the classical regression model in which rJ is known and
the nonparametric regression model Y == rJ (Z) + e with rJ an unknown smooth function. An
extension is to let the error distribution range over an infinite-dimensional set as well. D
25.3 Example (Logistic regression). Given a vec tor Z, let the random variable Y take
the value 1 with probability 1j(1 + e-r(Z)) and be O otherwise. Let Z == (ZI, Z2), and let
the function r be of the form r (z 1, Z2) == rJ (z 1) + eT Z2. Observed is the pair X == (Y, Z).
This is a semiparametric version of the logistic regres sion model, in which the response is
allowed to be nonlinear in part of the covariate. D
25.4 Example (Paired exponential). Given an unobservable variable Z with completely
unknown distribution, let X == (XI, X 2) be a vector of independent exponentially distributed
random variables with parameters Z and ze. The interest is in the ratio e of the conditional
hazard rates of X 1 and X 2. Modeling the "baseline hazard" Z as arandom variable rather
than as an unknown constant allows for heterogeneity in the population of all pairs (X 1, X 2 ),
and hence ensures a much better fit than the two-dimensional parametric model in which
the value z is a parameter that is the same for every observation. D
25.5 Example (Errors-in-variables). The observationis a pair X == (X 1, X 2 ), where XI ==
Z + e and X 2 == ex + f3Z + f for a bivariate normal vec tor (e, f) with mean zero and
unknown covariance matrix. Thus X 2 is a linear regres sion on a variable Z that is observed
with error. The distribution of Z is unknown. D
25.6 Example (Transformation regression). Suppose that X == (Y, Z), where the ran-
dom vectors Y and Z are known to satisfy rJ(Y) == eT Z + e for an unknown map 1J and
independent random vectors e and Z with known or parametrically specified distributions.
360
Semiparametric Models
The transformation 17 ranges over an infinite-dimensional set, for instance the set of all
monotone functions. O
25.7 Example (Cox). The observation is a pair X == (T, Z) of a "survival time" T and a
covariate Z. The distribution of Z is unknown and the conditional hazard function of T
given Z is of the form ee T Z A(t) for A being a completely unknown hazard function. The
parameter () has an interesting interpretation in terms of a ratio of hazards. For instance, if
the ith coordinate Zi of the covariate is a 0-1 variable then ee i can be interpreted as the ratio
of the hazards of two individuals whose covariates are Zi == 1 and Zi == O, respectively, but
who are identical otherwise. O
25.8 Example (Copula). The observation X is two-dimensional with cumulative distri-
bution function of the form Ce (G I (XI), G 2 (X2)), for a parametric family of cumulative
distribution functions Ce on the unit square with uniform marginals. The marginal distri-
bution functions G i may both be completely unknown or one may be known. O
25.9 Example (Frailty). Two survival times YI and Y 2 are conditionally independent
given variables (Z, W) with hazard function of the form W ee T zA (y). The random variable
W is not observed, possesses a gamma( 1), 1)) distribution, and is independent of the variable
Z which possesses a completely unknown distribution. The observation is X == (YI, Y 2 , Z).
The variable W can be considered an unobserved regression variable in a Cox model. O
25.10 Example (Random censoring). A "time of death" T is observed only if death oc-
curs before the time C of a "censoring event" that is independent of T; otherwise C is
observed. A typical observation X is a pair of a survival time and a 0-1 variable and
is distributed as (T J\ C, I{T < C}). The distributions of T and C may be completely
unknown. O
25.11 Example (Interval censoring). A "death" that occurs at time T is only observed to
have taken place or not at a known "check-up time" C. The observation is X == (C, 1 {T <
C}), and T and Care assumed independent with completely unknown or partially specified
distributions. O
25.12 Example (Truneation). A variable of interest Y is observed only if it is larger than
a censoring variable C that is independent of Y; otherwise, nothing is observed. A typical
observation X == (X 1, X 2 ) is distributed according to the conditional distribution of (Y, C)
given that Y > C. The distributions of Y and C may be completely unknown. O
25.2 Banach and HUbert Spaces
In this section we recall some facts conceming Banach spaces and, in particular, Hilbert
spaces, which play an important role in this chapter.
Given a probality space (X, A, P), we denote by L 2 (P) the set ofmeasurable functions
g : X f---* JR with P g2 == f g2 d P < 00, where almost surely equal functions are identi-
fied. This is a HUbert space, a complete inner-product space, relative to the inner product
25.2 Banach and Hilbert Spaces
361
and norm
(gl, g2) == Pg l g 2 ,
Ilgll == VI Pg2 .
Given a Hilbert space JHI, the projection lemma asserts that for every g E IHI and convex,
closed subset C c JHI, there exists a unique element I1g E C that minimizes c r---+ II g-c II
over C. If C is a closed, linear subspace, then the projection I1g can be characterized by
the orthogonality relationship
(g - TIg, c) == O,
every c E C.
The proof is the same as in Chapter 11. If Cl c C 2 are two nested, closed subspaces,
then the projection onto Cl can be found by first projecting onto C 2 and next onto Cl.
Two subsets Cl and C 2 are orthogonal, notation C 1- C 2 , if (Cl, C2) == O for every pair of
Ci E C i . The projection onto the sum of two orthogonal closed subspaces is the sum of the
projections. The orthocomplement C.1. of a set C is the set of all g 1- C.
A Banach space is a complete, normed space. The dual space Iffi* of a Banach space Iffi
is the set of all continuous, line ar maps b* : Iffi r---+ JR. Equivalently, all line ar maps such that
Ib*(b)1 < Ilb*lllIbll for every b E Iffi and some number IIb*ll. The smallest number with
this property is denoted by II b* II and defines a norm on the dual space. According to the
Riesz representation theorem for Hilbert spaces, the dual of a Hilbert space lHI consists of
all maps
h r---+ (h, h *) ,
where h * ranges over IHI. Thus, in this case the dual space JHI* can be identified with the space
1HI itself. This identification is an isometry by the Cauchy-Schwarz inequality I (h, h*) I <
Ilhllllh*lI.
A linear map A : Iffi l B 2 from one Banach space into another is continuous if and only
if "Ab l ll 2 < II A 1111 bIli for every bI E Iffi l and some number II A II. The smallest number
with this property is denoted by II A II and defines a norm on the set of all continuous, line ar
maps, also called operators, from BI into Iffi 2 . Continuous, linear operators are also called
"bounded," even though they are only bounded on bounded sets. To every continuous, line ar
operator A : JESI r---+ Iffi 2 exists an adjoint map A * : JES; Iffi7 defined by (A * b )bl == b AbI.
This is a continuous, linear operator of the same norm "A * II == II A II. For Hilbert spaces the
dual space can be identified with the original space and then the adjoint of A : IHII IHI 2 is
a map A * : JHI 2 IHII. It is characterized by the property
(AhI, h 2 )2 == (hI, A*h 2 )1,
every hI E IHI l , h 2 E IHI 2 .
An operator between Euclidean spaces can be identified with a matrix, and its adjoint with
the transpose. The adjoint of a restriction Aa : IHII,a C IHII r---+ IHI 2 of A is the composition
TI o A * of the projection TI : IHIl r---+ IHIl,a and the adjoint of the original A.
The range R(A) == {AbI: bI E JESI} of a continuous, linear operator is not necessarily
closed. By the "bounded inverse theorem" the range of a 1-1 continuous, line ar operator
between Banach spaces is closed if and only if its inverse is continuous. In contrast the
kemel N(A) == {bI: AbI == O} of a continuous, linear operator is always closed. For an
operator between Hilbert spaces the relationship R(A).1. == N(A *) follows readily from the
362
Semiparametric Models
characterization of the adjoint. The range of A is closed if and only if the range of A * is
closed if and only if the range of A * A is closed. In that case R (A *) == R (A * A).
If A * A : IHI 1 r--+ IHI 1 is continuously invertible (i.e., is 1-1 and onto with a continuous
inverse), then A(A * A)-l A * : IHI 2 r--+ R(A) is the orthogonal projection onto the range of A,
as follows easily by checking the orthogonality relationship.
25.3 Tangent Spaces and Information
Suppose that we observe arandom sample XI, . . . , X n from a distribution P that is known
to belong to a set P of probability measures on the sample space (X, A). It is required to
estimate the value 1/1' (P) of a functional 1/1' : P r--+ }Rk. In this section we develop a notion
of information for estimating 1/1' (P) given the model P, which extends the notion of Fisher
information for parametric models.
To estimate the parameter 1/1' (P) given the model P is certainly harder than to estimate
this parameter given that P belongs to a submodel Po c P. For every smooth parametric
submodel Po == {Po: () E e} eP, we can calculate the Fisher information for estimating
1/1' (Po). Then the information for estimating 1/1' (P) in the whole model is certainly not
bigger than the infimum of the informations over all submodels. We shall simply define
the information for the whole model as this infimum. A submodel for which the infimum
is taken (if there is one) is called least favorable or a "hardest" submodel.
In most situations it suffices to consider one-dimensional submodels Po. These should
pass through the "true" distribution P of the observations and be differentiable at P in the
sense of Chapter 7 on local asymptotic normality. Thus, we consider maps t r--+ Pf from a
neighborhood of O E [O, ex)) to P such that, for some measurable function g : X 1--+ JR, t
f [ dP1/2 - dP1/2 1 ] 2
f _ _gdP1/2 O.
t 2
(25.13)
In other words, the parametric submodel {Pf : O < t < 8} is differentiable in quadratic
mean at t == O with score function g. Letting t 1--+ Pf range over a collection of submodels,
we obtain a collection of score functions, which we call a tangent set of the model P at P
and denote by P p . Because Ph 2 is automatically fini te, the tangent space can be identified
with a subset of L 2 (P), up to equivalence classes. The tangent set is often a linear space,
in which case we speak of a tangent space.
Geometrically, we may visualize the model P, or rather the corresponding set of "square
roots of measures" d P 1/ 2 , as a subset of the unit ball of L 2 (P), and P p, or rather the set of
all objects g dP1/2, as its tangent set.
U sually, we construct the submodels t 1--+ Pt such that, for every x,
a
g(x) == - logdPf(x).
at It=O
t If P and every one of the measures Pt possess densities p and Pt with respeet to a meas ure Mt, then the
expressions d P and d Pt can be replaeed by p and Pt and the integral can be understood relative to Mt (add d Mt
on the right). We use the notations d Pt and d P, beeause some model s P of interest are not dominated, and the
ehoiee of Mt is irrelevant. However, the model eould be taken dominated for simplieity, and then d Pt and d P
are just the densities of Pt and P.
25.3 Tangent Spaces and Informatian
363
However, the differentiability (25.13) is the correct definition for defining information,
because it ensures a type of local asymptotic normality. The following lemma is proved in
the same way as Theorem 7.2.
25.14 Lemma. Ifthe path t f--* Pt in P satisfies (25.13), then Pg == O, Pg2 < 00, and
O n dP1/.Jii 1 1 2
log (Xi) == g(Xi) - -Pg + op(l).
i=1 dP yn i=1 2
For defining the information for estimating ljf(P), only those submodels t f--* Pt along
which the parameter t f--* ljf(Pt) is differentiable are of interest. Thus, we consider only
submodels t f--* Pt such that t f--* ljf (Pt) is differentiable at t == O. More precisely, we
define ljf : P f--* IR k to be differentiable at P relative to a given tangent set P p if there exists
a continuous linear map f p : L 2 (P) 1--+ IR k such that for every g E P p and a submodel
t f--* Pt with score function g,
1/f ( P t ) - ljf ( p ) .
-+ ljfpg.
t
This requires that the derivative of the map t f--* ljf (Pt) exists in the ordinary sense, and also
that it has a special representation. (The map ljf p is much like a Hadamard derivative of ljf
viewed as a map on the space of "square roots of measures.") Our definition is also relative
to the submodels t f--* P t , but we speak of "relative to P p" for simplicity.
By the Riesz representation theorem for Hilbert spaces, the map ljf p can always be
written in the form of an inner product with a fixed vector-valued, measurable function
- k
ljf p : Xf--* IR ,
pg= (]frp,g)p = f ]frpgdP.
Here the function 1jr p is not uniquely defined by the functionalljf and the model p, because
only inner products of 1f; p with elements of the tangent set are specified, and the tangent set
does not span all of L2(P). However, it is always possible to find a candidate VJ p whose
coordinate functions are contained in lin P p, the closure of the linear span of the tangent
set. This function is unique and is called the efficient in.fluence function. It can be found as
the projection of any other "influence function" onto the closed linear span of the tangent
set.
In the preceding set-up the tangent sets P p are made to depend both on the model P and
the functionalljf. We do not always want to use the "maximal tangent set," which is the set
of all score functions of differentiable submodels t f--* P t , because the parameter 1/1 may not
be differentiable relative to it. We consider every subset of a tangent set a tangent set itself.
. .
The maximal tangent set is acone: If g E P p and a > O, then ag EPp, because the
path t f--* Pat has score function ag when t f--* Pt has score function g. It is rarely a loss of
generality to assume that the tangent set we work with is a cone as well.
25.15 Example (Parametric model). Consider a parametric model with parameter () rang-
ing over an open subset e of]Rm given by densities pe with respect to some measure /-1.
Suppose that there exists a vector-valued measurable map £e such that, as h -+ O,
[ ] 2
1/2 ] /2 1 T' 1/2 2
f Pe+h - Pe - 2 h le Pe dfL = o(lIh II ).
364
Semiparametric Models
Then a tangent set at Pe is given by the linear space {hT'€e : h E m} spanned by the score
functions for the coordinates of the parameter e .
If the Fisher information matrix le == Pelel is invertible, then every map X : 0) 1---+ }Rk
that is differentiable in the ordinary sense as a map between Euclidean spaces is differentiable
as a map 1/1 (Pe) == X (e) on the model relative to the given tangent space. This follows
because the submodel t 1---+ P e + th has score h T le and
x (e + th) = Xe h = Pe [ (xe le-l .€e) h T .€e ] .
ati t =0
This equation shows that the function 1/1 Pe == Xe 1;;1 .€e is the efficient influence function.
In view of the resuits of Chapter 8, asymptotically efficient estimator sequences for X ce)
are asymptotically linear in this function, which justifies the name "efficient influence
function." O
25.16 Example (Nonparametric models). Suppose that P consists of all probability
laws on the sample space. Then a tangent set at P consists of all measurable functions g
satisfying f g dP == O and f g2 dP < 00. Because a score function necessarily has mean
zero, this is the maximal tangent set.
It suffices to exhibit suitable one-dimensional submodels. For a bounded function g,
consider for instance the exponential family Pt(x) == c(t) exp(tg(x)) po(x) or, alterna-
tively, the model Pt (x) == (1 + t g (x) ) PO (x). Both models have the property that, for every
x,
a
g(x) == - log Pt(x).
at It=O
By a direct calculation or by using Lemma 7.6, we see that both models also have score
function g at t == O in the L 2 -sense (25.13). For an unbounded function g, these submodels
are not necessarily well-defined. However, the models have the common structure Pt(x) ==
c(t) k(tg(x)) po(x) for a nonnegative function k with k(O) == k'(O) == 1. The function
k(x) == 2(1 + e- 2x )-1 is bounded and can be used with any g. O
25.17 Example (Cox model).\ The density of an observation in the Cox model takes the
form
(t, z) 1---+ e-eeTZA(t) A(t) eeTz pz(z).
Differentiating the logarithm of this expression with respect to e gives the score function
for e,
z - zeeTZA(t).
We can also insert appropriate parametric models s 1---+ As and differentiate with respect to
s. If a is the derivative of log As at s == O, then the corresponding score for the model for
the observation is
a(t) - eeTz l adA.
[O,t]
Finally, scores for the density pz are functions bez). The tangent space contains the linear
span of all these functions. Note that the scores for Acan be found as an "operator" working
on functions a. O
25.3 Tangent Spaces and Informatian
365
25.18 Example (Transformation regression model). If the transformation rJ is increas-
ing and e has density ep, then the density of the observation can be written ep ( rJ (y) -
eT z) rJ'(y) pz(z). Scores for e and TJ take the forms
ep'
- z -;;; ( 17 (y) - eT z),
ep' a'
-(rJ(y) - eT Z ) a(y) + -(y),
ep ry'
where a is the derivative for TJ. If the distributions of e and Z are (partly) unknown, then
there are additional score functions corresponding to their distributions. Again scores take
the form of an operator acting on a set of functions. O
To motivate the definition of information, assume for simplicity that the parameter ljJ (P)
is one-dimensiona1. The Fisher information about t in a submodel t 1-+ Pt with score function
g at t == O is P g2. Thus, the "optimal asymptotic variance" for estimating the function
t 1/1 (Pt), evaluated at t == O, is the Cramer-Rao bound
( d ljJ ( Pt ) / d t ) 2
Pg2
- 2
(ljJp, g)p
(g, g) p
The supremum of this expression over all submodels, equivalently over all elements of the
tangent set, is a lower bound for estimating ljJ (P) given the model P, if the "true measure"
is P. This supremum can be expressed in the norm of the efficient influence function 1f p .
25.19 Lemma. Suppose that the functional ljJ : P 1-+ JR is differentiable at P relative to
the tangent set P p . Then
- 2
(1/Jp, g)p P '171'2
sup == p'
gElin PP (g, g) p
Proof. This is a consequence of the Cauchy-Schwarz inequality (P1fr pg)2 < P1fr P g2
and the fact that, by definition, the efficient influence function 1f p is contained in the closure
of lin P p . .
Thus, the squared norm p 1f of the efficient influence function plays the role of an
"optimal asymptotic variance," just as does the expression (3I(3-1 J in Chapter 8. Similar
considerations (take linear combinations) show that the "optimal asymptotic covariance" for
estimating a higher-dimensional parameter ljJ : P 1-+ JRk is given by the covariance matrix
p 1f p 1/1 of the efficient influence function.
In Chapter 8, we developed three ways to give a precise meaning to optimal asymptotic
covariance: the convolution theorem, the almost -everywhere convolution theorem, and
the mini max theorem. The almost-everywhere theorem uses the Lebesgue measure on the
Euclidean parameter set, and does not appear to have an easy parallel for semiparametric
models. On the other hand, the two other results can be generalized.
For every g in a given tangent set P p , write Pt,g for a submodel with score function g
along which the function ljJ is differentiable. As usual, an estimator Tn is a measurable
function Tn (X 1, . . . , X n ) of the observations. An estimator sequence Tn is called regular
at P for estimating ljJ(P) (relative to P p ) if there exists a probability measure L such that
C ( ) Pl/.jn,g
yn 1;1 -ljJ(P 1j y'n,g) 'v'7 L,
every g EPp.
366
Semiparametric Models
25.20 Theorem (Convolution). Let the function 1/J : p IR k be differentiable at P rel-
ative to the tangent cone P p with efficient injluence function 1f p. Then the asymptotic
covariance matri of every regular sequence of estimators is bounded below by P1f P1f .
Furthermore, if p p is a convex cone, then every limit distribution L of a regular sequence
of estimators can be written L == N(O, p1f P1f ) * M for some probability distribu-
tion M.
25.21 Theorem (LAM). Let the function 1/J : p IR k be differentiable at P relative to
the tangent cone P p with efficient injluence function 1f p. IfP p is a convex cone, then, for
any estimator sequence {Tn} and subconvexfunction l : IR k [O, (0),
sup l f sup E Pl/,,",g € ( .Jn (Tn - 1/r (PIj ..jii,g)) )) > f l d N (O, p 1f p 1f ).
I gEI
Here the first supremum is taken over all finite subsets I of the tangent set.
Proofs. These results folIow essentialIy by applying the corresponding theorems for para-
metric models to sufficiently rich finite-dimensional submodels. However, because we have
defined the tangent set using one-dimensional submodels t Pt,g, it is necessary to rework
the proofs a little.
Assume first that the tangent set is a linear space, and fix an orthonormal base g p ==
(gl,..., gm)T ofanarbitraryfinite-dimensionalsubspace.Forevery g Elin gp selectasub-
model t 1---+ Pt,g as used in the statement of the theorems. Each of the submodels t 1---+ Pt,g is
localIy asymptoticalIy normal at t == O by Lemma 25.14. Therefore, because the covariance
matrix of g p is the identity matrix,
(P;/ ,hT gp : h E JRm) 'v'7 (N m (h, I) : h E JRm)
in the sense of convergence of experiments. The function 1/Jn (h) == 1/J CP1/-Jn,hT gp) satisfies
.Jn(1/Jn(h) -1/Jn(O)) -+ phT gp == (P1fpg )h == : Ah.
For the same Ck x m) matrix the function Ag p is the orthogonal projection of 1f p onto
lin g p, and it has covariance matrix AA T. Because 1f p is, by definition, contained in the
closed linear span of the tangent set, we can choose g p such that 1f p is arbitrarily close to
its projection and hence AA T is arbitrarily close to P 1f p 1f .
U nder the assumption of the convolution theorem, the limit distribution of the sequence
Vfi(Tn - 1/Jn (h)) under P1/-Jn,hT gp is the same for every h E JRm. By the asymptotic
representation theorem, Proposition 7.10, there exists a randomized statistic T in the limit
experiment such that the distribution of T - Ah under h does not depend on h. By
Proposition 8.4, the nulI distribution of T contains a normal N (O, AA T)-distribution as a
convolution factor. The proof of the convolution theorem is complete upon letting AA T
- - T
tend to P 1/J p 1/J p .
Under the assumption that the sequence Vfi(Tn -1/J(P)) is tight, the minimax theorem is
proved similarly, by first bounding the left side by the minimax risk relative to the submodel
corresponding to gp, and next applying Proposition 8.6. The tightness assumption can be
dropped by a compactification argument. (see, e.g., [139], or [146]).
If the tangent set is a convex cone but not a linear space, then the submodel constructed
previously can only be used for h ranging over a convex cone in IR m . The argument can
25.3 Tangent Spaces and Information
367
remain the same, except that we need to replace Propositions 8.4 and 8.6 by stronger resu1ts
that refer to convex cones. These extensions exist and can be proved by the same Bayesian
argument, now choosing priors that flatten out ins ide the cone (see, e.g., [139]).
If the tangent set is a cone that is not convex, but the estimator sequence is regular, then
we use the fact that the matching randomized estimator T in the limit experiment satisfies
EhT == Ah + EoT for every eligible h, that is, every h such that h T gp EPp. Because
the tangent set is acone, the latter set includes parameters h == t hi for t > O and directions
hi spanning JRm. The estimator T is unbiased for estimating Ah + Eo T on this parameter
set, whence the covariance matrix of T is bounded below by AA T, by the Cramer-Rao
inequality. .
Both theorems have the interpretation that the matrix P 1f p 1f is an optimal asymptotic
covariance matrix for estimating 'ljJ(P) given the model P. We might wish that this could
be formulated in asimpler fashion, but this is precluded by the problem of superefficiency,
as is already the case for the parametric analogues, discussed in Chapter 8. That the notion
of asymptotic efficiency us ed in the present interpretation should not be taken absolutely is
shown by the shrinkage phenomena discussed in section 8.8, but we use it in this chapter.
We shaH say that an estimator sequence is asymptotically efficient at P, if it is regular at P
with limit distribution L == N(O, p1f P1f ).t
The efficient influence function 1f p plays the same role as the normalized score function
Ie- I .ee in parametric models. In particular, a sequence of estimators Tn is asymptotically
efficient at P if
1 n
0i(Tn -1jf(P)) = Jn P(Xi) +Op(1).
(25.22)
This justifies the name "efficient influence function."
25.23 Lemma. Let the function ljJ : P }Rk be differentiable at P relative to the tangent
cone p p with efficient in.fluence function 1f p. A sequence of estimators Tn is regular at P
with limiting distribution N(O, P1f P1f ) if and only ifit satisfies (25.22).
Proof. Because the submodels t Pt,g are locally asymptotically normal at t == O, "if"
follows with the help of Le Cam's third lemma, by the same arguments as for the analogous
resu1t for parametric models in Lemma 8.14.
To prove the necessity of (25.22), we adopt the notation of the proof of Theorem 25.20.
The statistics Sn == 1fr(P) + n-I "L7=11f p(X i ) depend on P but can be considered a true
estimator sequence in the local subexperiments. The sequence Sn trivially satisfies (25.22)
and hence is another asymptotically efficient estimator sequence. We may assume for sim-
plicity that the sequence Jn(Sn -1fr(PI/-Jn,g), Tn - 'ljJ(P I /-Jn,g)) converges under every
local parameter g in distribution. Otherwise, we argue along subsequences, which can be
t If the tangent set is not a linear space, then the situation beeomes even more eomplieated. If the tangent set is
a eonvex cone, then the minimax risk in the left side of Theorem 25.21 cannot fall below the normal risk on
the right side, but there may be nonregular estimator sequenees for which there is equality. If the tangent set is
not eonvex, then the assertion of Theorem 25.21 may fail. Convex tangent eones arise frequently; fortunately,
noneonvex tangent eones are rare.
368
Semiparametric Models
selected with the help of Le Cam's third lemma. By Theorem 9.3, there exists a match-
ing randornized estimator (S, T) == (S, T)(X, U) in the normailirnit experiment. By the
efficiency of both sequences Sn and Tn, the variables S - Ah and T - Ah are, under h,
marginally normally distributed with mean zero and covariance matrix P {jr p 1/r . In partic-
ular, the expectations Eh S == Eh Tare identically equal to Ah. Differentiate with respect to
h at h == O to find that
EoSXT == EoTXT == A.
It follows that the orthogonal projections of S and T onto the linear space spanned by the
coordinates of X are identical and given by ns == nT == AX, and hence
Covo(S - T) == Covo(n-L S - n-LT) < 2 Covo n-L S + 2 Covo n-LT.
(The inequality means that the difference of the matrices on the right and the left is
nonnegative-definite.) We have obtained this for a fixed orthonormal set gp == (gl, . . . , gm).
If we choose gp such that AA T is arbitrarily close to P1/r P1/r , equivalently Covo TIT ==
AA T == Covo ns is arbitrarily close to Covo T == P1/r P1/r == Cova S, and then the right
side of the preceding display is arbitrarily close to zero, whence S - T O. The proof is
complete on noting that -Jn,(Sn - Tn) S - T. .
25.24 Example (Empirical distribution). The empirical distribution is an asymptotically
efficient estimator if the underlying distribution P of the sample is completely unknown.
To give a rigorous expression to this intuitively obvious statement, fix a measurable func-
tion f : X f---+ IR with P f2 < 00, for instance an indicator function f == 1 A, and consider
JID n f == n-I 2::7=1 f (Xi) as an estimator for the function 1/1 (P) == P f.
In Example 25.16 it is seen that the maximal tangent space for the nonparametric model is
equal to the set of all g E L 2 (P) such that P g == O. For a general function f, the parameter 1/1
may not be differentiable relative to the maximal tangent set, but it is differentiable relative
to the tangent space consisting of all bounded, measurable functions g with P g == O. The
closure of this tangent space is the maximal tangent set, and hence working with this smaller
set does not change the efficient influence functions. For a bounded function g with P f == O
we can use the submodel defined by dP t == (1 + 19) dP, for which 1/1 (Pt) == Pf + tPfg.
Renee the derivative of 1/1 is the map g f---+ 1/1 p g == P f g, and the efficient influence function
relative to the maximum tangent set is the function 1/r p == f - P f. (The function f is an
influence function; its projection onto the mean zero functions is f - p f.)
The optimal asymptotic variance for estimating P f---+ P f is equal to P 1/r == p (f -
P f)2. The sequence of empirical estimators JID n f is asymptotically efficient, because it
satisfies (25.22), with the o p (l)-remainder term identically zero. D
25.4 Efficient Score Functions
A function 1/1 (P) of particular interest is the parameter e in a serniparametric model
{Pe,rJ : e E 8, 77 EH}. Rere 8 is an open subset of IR k and H is an arbitrary set, typically
of infinite dimension. The information bound for the functional of interest 1/1 (P e , rJ) == e can
be conveniently expressed in an "efficient score function."
25.4 Efficient Score Functions
369
As submodels, we use paths of the form t f---+ Pe+ta,rJt' for given paths t f---+ rJt in the
parameter set H. The score functions for such submodels (if they exist) typically have the
form of a sum of "p arti al derivatives" with respect to e and rJ o If f e , rJ is the ordinary score
function for e in the model in which rJ is fixed, then we expect
a T'
- log d Pe+ta,rJt == a fe,rJ + g.
at It = o
The function g has the interpretation of a score function for rJ if e is fixed and mns through
an intinite-dimensional set if we are concerned with a "true" semiparametric model. We
refer to this set as the tangent set for rJ, and den ote it by rJ P Pe,1] .
The parameter 1/.I(P e + ta ,rJt) == e + ta is certainly differentiable with respect to t in the
ordinary sense but is, by definition, differentiable as a parameter on the model if and only
if there exists a function 1jJ e, rJ such that
a - T'
a == - a 1/.1 (Pe+ta,rJ) == (1/.Ie n, a fe,rJ + g)P e 1] '
tit =0 "I ,
k .
a E IR , g E rJ P P e ,1] .
Setting a == O, we see that 1jf e,rJ must be orthogonal to the tangent set rJ Pp e , 1] for the nuisance
p ameter. Define f1 e ,rJ as the orthogonal projection onto the closure of the linear span of
rJPPe,1J in L 2 (P e ,rJ)'
The function defined by
le,rJ == fe,rJ - f1e,rJfe,rJ
is called the efficient score function for e, and its covariance matrix 1 e,rJ == Pe'rJle'rJl 'rJ is
the efficient information matrix.
25.25 Lemma. Suppose that for every a E IR k and every g E rJ P P e ,1] there exists a path
t f---+ 1]t in H such that
f [ dPl/2 - dPl/2 1 ] 2
O+ta, tt e, -"2 (aT ie, + g) dPi, 2 --7 O.
(25.26)
If 1 e,rJ is nonsingular, then the functional 1/1 (Pe,rJ) == e is differentiable at Pe,rJ relative to
. . . - --1 -
the tangent set PPe,1) == lin fe,rJ + rJ Pp e , I) with efficient influence function 1/.1 e,rJ == I e,rJfe,rJ o
ProoJ. The given set P Pe,1J is a tangent set by assumption. The function 1/1 is differentiable
with respect to this tangent set because
(l; le'rJ' aT fe,rJ + g) e == l; (le'rJ' f 'rJ) e a ==a.
,1] ,1J
The last equality follows, because the inner product of a function and its orthogonal
projection is equal to the square length of the projection. Thus, we may replace fe,rJ by
le,rJ' .
Consequently, an estimator sequence is asymptotically efficient for estimating e if
r:: 1 --1-
V n(Tn - e) == vn I e,rJfe,rJ(Xi) + oPe,1) (1).
n, 1
l=
370
Semiparametric Models
This equation is very similar to the equation derived for efficient estimators in parametric
models in Chapter 8. It differs only in that the ordinary score function Re,TJ has been replaced
by the efficient score function (and similarly for the information). The intuitive explanation
is that a part of the score function for e can also be accounted for by score functions for the
nuisance parameter 17. If the nuisance parameter is unknown, a part of the information for
e is "lost," and this corresponds to a loss of a part of the score function.
25.27 Example (Symmetric location). Suppose that the model consists of all densities
x 1---+ 17(x - e) with e E JR and the "shape" 17 symmetric about O with finite Fisher informat-
ion for location I TJ' Thus, the observations are sampled from a density that is symmetric
about e.
By the symmetry, the density can equivalently be written as 17 (Ix - el). It follows that
any score function for the nuisance parameter 17 is necessarily a function of Ix - e I. This
suggests a tangent set containing functions of the fonn a(17 1 /17)(x - e) + b(lx - el). It is
not hard to show that all square- integrable functions of this type with mean zero occur as
score functions in the sense of (25.26). t
A symmetric density has an asymmetric derivative and hence an asymmetric score func-
tion for location. Therefore, for every b,
17 1
Ee,TJ - (X - e) b(IX - el) == O.
17
Thus, the projection of the e -score onto the set of nuisance scores is zero and hence the
efficient score function coincides with the ordinary score function. This means that there
is no difference in infonnation about e whether the fonn of the density is known ar not
known, as long as it is known to be symmetric. This surprising fact was discovered by Stein
in 1956 and has been an important motivation in the early work on semiparametric models.
Even more surprising is that the infonnation calculation is not misleading. There exist
estimator sequences for e whose definition does not depend on 17 that have asymptotic
variance 1;;1 under any tru 17. See section 25.8. Thus a symmetry point can be estimated
as well if the shape is known as if it is not, at least asymptotically. O
25.28 Example (Regression). Let ge be a given set of functions indexed by a parameter
e E JRk, and suppose that a typical observation (X, Y) follows the regression model
Y == ge(X) + e,
E(e I X) ==0.
This model includes the logistic regression model, for ge (x) == 1/( 1 + e-e T x). It is also
a version of the ordinary linear regression model. However, in this example we do not
assume that X and e are independent, but only the relations in the preceding display, apart
from qualitative smoothness conditions that ensure existence of score functions, and the
existence of moments. We shalI write the fonnulas assuming that (X, e) possesses a density
17. Thus, the observation (X, Y) has a density 17(X, y - ge(x)), in which 17 is (essentially)
only restricted by the relations f e 17 (x, e) de - O.
Because any perturbation 17t of 17 within the model must satisfy this same relation
f e17t (x, e) de == O, it is clear that score functions for the nuisance parameter 17 are functions
t That no other funetions can oeeur is shown in, for example, [8, p. 56-57] but need not concem us here.
25.5 Score and Information Operators
371
a (x, y - ge (x)) that satisfy
( ) _ f ea(X, e) rJ(X, e) de _
E ea(X, e) IX - J - O.
rJ(X, e) de
By the same argument as for nonparametric models all bounded square-integrable functions
of this type that have mean zero are score functions. Because the relation E (ea (X, e) IX) == O
is equivalent to the orthogonality in L 2 (rJ) of a(x, e) to all functions of the form eh(x), it
follows that the set of score functions for rJ is the orthocomplement of the set eH, of all
functions of the form (x, y) f-+ (y - ge(x))h(x) within L 2 (P e ,17)' up to centering at mean
zero.
Thus, we obtain the efficient score function for e by projecting the ordinary score function
le,17 (x, y) == -rJ2/rJ (x, e)ge (x) onto eH. The projection of an arbitrary function b(x, e) onto
the functions eH is a function eha(x) such that Eb(X, e)eh(X) == Eeha(X)eh(X) for all
measurable functions h. This can be solved for ha to find that the projection operator takes
the fonn
E(b(X, e)e I X)
TIeHb(X, e) == e 2 .
E(e IX)
This readily yields the efficient score function
- ege (X) f rJ2(X, e)e de (Y - ge (X) )ge (X)
l e ( X Y ) - - -
,17 ' - E(e 2 IX) f rJ(X, e) de - E(e 2 IX) .
The efficient information takes the form I e,17 == E(gegJ (X)/E(e 2 1 X)). O
25.5 Score and Information Operators
The method to find the efficient influence function of a parameter given in the preceding
section is the most convenient method if the model can be naturally partitioned in the
parameter of interest and a nuisance parameter. For many parameters such a partition
is impossible or, at least, unnatural. Furthermore, even in semiparametric models it can
be worthwhile to derive amore concrete description of the tangent set for the nuisance
parameter, in terms of a "score operator."
Consider first the situation that the model P == {P 17 : rJ E H} is indexed by a parameter
rJ that is itself a probability measure on some measurable space. We are interested in
estimating a parameter of the type 1/J'(P 17 ) == X (rJ) for a given functiorr X : H }Rk on the
model H.
The model H gives rise to a tangent set H 17 at rJ. If the ma rJ f-+ P 17 is differentiable in an
appropriate sense, then its derivative maps every score b E H 17 into a score g for the model
P. To make this precise, we assume that a smooth parametric submodel t rJt induces a
smooth parametric submodel t P 17t , and that the score functions b of the submodel t rJt
and g of the submodel t PTJt are related by
g == ATJb.
Then ATJ H TJ is a tangent set for the model P at PTJ' Because ATJ turns scores for the model H
into scores for the model P it is called aseore operator. It is seen subsequently here that if rJ
372
Semiparametric Models
and P17 are the distributions of an unobservable Y and an observable X == m (Y), respectively,
then the score operator is a conditional expectation. More generally, it can be viewed as a
derivativeofthemap17 1---+ P17' WeassumethatA17,asamapA17: lin H17 C L 2 (17) 1---+ L2(P17)'
is continuous and linear.
N ext, assume that the f nction 17 1---+ X (17) is differentiable with influence function X17
relative to the tangent set H 17' Then, by d finition, he function 1/1 (P?}) == X (17) is pathwise
differentiable relative to the tangent set P P,] == A17 H 17 if and only if there exists a vector-
valued function 1if P such that
1)
/- ) a a
\1/JpT},A17b p ==- a 1/J(p?}J==- a X(17t)==(X17,b)?},
T} tlt=O tlt=O
b EH?}.
This equation can be rewritten in terms of the adjoint score operator A : L 2 (P?}) 1---+ lin H 17'
By definition this satisfies (h, A?}b) PT} == (A h, b)17 for every h E L2(P17) and b E iI 17.t The
preceding display is equivalent to
A 1if P,] == X17'
(25.29)
We conclude that the function 1/J(p?}) == X (r]) is differentiable relative to the tangent set
p PT} == A17 iI 17 if and only if this equation can be solved for 1f PT}; equivalently, if and onl if
X17 is contained in the range of the adjoint A . Because A is not necessarily onto lin H 17'
not even if it is one-to-one, this is a condition.
For multivariate functional s (25.29) is to be understood coordinate-wise. Two solutions
1if p of (25.29) can differ only by an element of the kernel N(A ) of A , which is the
T} .
orthocomplement R(A17)l. of the range of A?} : lin H?} 1---+ L 2 (P?}). Thus, there is at most
one solution 1if P1) that is contained in R (A?}) == lin A17iI 17' the closure of the range of A17' as
required.
If X17 is contained in the smaller range of A A17' then (25.29) can be solved, of course,
and the solution can be written in the attractive fonn
1if PT} == A?} (A A17) - X17'
(25.30)
Here A A?} is called the information operator, and (A A?})- is a "generalized inverse."
(Here this will not mean more than that b == (A A?})- X17 is a solution to the equation
A A?}b == X17.) In the preceding equation the operator A A17 performs a similar role as
the matrix X T X in the least squares solution of a linear regression model. The operator
A?}(A A17)-l A (if it exists) is the orthogonal projection onto the range space of A?}.
So far we have assumed that the parameter 17 is a probability distribution, but this is not
necessary. Consider the more general situation of a model P == {P?} : 17 E H} indexed by a
parameter 17 running through an arbitrary set H. Let 1HI?} be a subset of a Hilbert space that
indexes "directions" b in which 17 can be approximated within H. Suppose that there exist
continuous, linear operators A17: lin 1HI?}1---+ L2(P17) and X17: lin IHI17 1---+ }Rk, and for every
b E IHI17 a path t 1---+ 17t such that, as t t O,
/ [ dP1/2 _ dP1/2 1 ] 2
17( 17 _ _ A b d pl 12 O
t 2 17 17 '
X(17t)-X(17)
X17 b .
t
t Note that we de fine A to have range lin iI 1]' so that it is the adjoint of A1] : iI 1] 1---+ L 2,C P 17 ). This is the adjoint
of an extension A1] : L2(17) 1---+ L2(P1]) followed by the orthogonal projeetion onto lin H 1]'
25.5 Score and Informatian Operators
373
By the Riesz representation theorem for Hilbert spaces, the "derivative" XrJ has a repre-
sentation as an inner product X17b == (X 1 7' b)IHII) for an element XrJ E lin lHI . The preceding
discussion can be extended to this abstract set-up.
25.31 Theorem. The map 1/1 : p r-+ IR k given by 1/1 (PrJ) == X (1]) is differentiable at PrJ
relative to the tangent set ArJlliI17 if and only if each coordinate function of XrJ is contained in
the range of A : L 2 (PrJ) r-+ lin lHIrJ' The efficient influence function 1f PI ) sati sfies (25.29).
If each coordinate function of XrJ is contained in the range of A ArJ : lin lliIrJ f--* lin lliIrJ' then
it also satisfies (25.30).
Proof. By assumption, the set ArJlliIrJ is a tangent set. The map 1/1 is differentiable relative
to this tangent set (and the corresponding submodels t f--* P rJt) by the argument leading up
to (25.29). .
The condition (25.29) is odd. By definition, the influence function XrJ is contained in
the closed linear span of 1HIrJ and the operator A maps L 2 (PrJ) into lin 1HIrJ. Therefore, the
condition is certainly satisfied if A is onto. There are two rea sons why it may fail to be
onto. First, its range R(A ) may be a pro per subspace of lin lHIrJ' Because b ...L R(A )
if and only if b E N(ArJ)' this can happen only if ArJ is not one-to-one. This means that
two different directions b may lead to the same score function ArJb, so that the information
matrix for the corresponding two-dimensional submodel is singular. A rough interpretation
is that the parameter is not locally identifiable. Second, the range space R(A ) may be
dense but not closed. Then for any XrJ there exist elements in R(A ) that are arbitrarily
close to X17' but (25.29) may still faiI. This happens quite often. The following theorem
shows that failure has serious consequences. t
25.32 Theorem. Suppose that 1] f--* X (1}) is differentiable with influence function XrJ ti-
R(A ). Then there exists no estimator sequence for X (1}) that is regular at Pr]"
25.5.1 Semiparametric Models
In a semiparametric model {Pf), rJ : e E e, 17 EH}, the pair (e, 17) plays the role of the single
1] in the preceding general discussion. The two parameters can be perturbed independently,
and the score operator can be expected to take the form
T'
Af),rJ(a, b) == a ff),rJ + Bf),rJ b .
Here Bf), rJ : 1HIrJ r-+ L 2 (Pf), 17) is the score operator for the nuisance parameter. The domain of
the operator Af), rJ : R k x lin lliI17 f--* L 2 (Pf), rJ) is a Hilbert space relative to the inner product
( (a, b), (ex, fJ)) 1] == aT ex + (b, fJ) IHII) .
Thus this example fits in the general set-up, with IR k x IHI1] playing the role of the earlier
1HIrJ' We shalI derive expressions for the efficient inftuence functions of e and 1].
The efficient inftuence function for estimating e is expressed in the efficient score function
for e in Lemma 25.25, which is defined as the ordinary score function minus its projection
t For a proof, see [140].
374
Semiparametric Models
onto the score-space for 1]. Presently, the latter space is the range of the operator Be,r]. If
the operator B;,r]Be,r] is continuously invertible (but in many examples it is not), then the
operator Be,r] (B;,r] B e,r])-l B;,r] is the orthogonal projection onto the nuisance score space,
and
le,r] == (I - Be,r] (B;,r]B e ,r])-l B;,r])fe,,,.
(25.33 )
This means that b ==-(B;,,,B e ,r])-l B;,,,.€e,,, is a "Ieast favorable direction" in H, for esti-
mating e. If e is one-dimensional, then the submodel t r-+ Pe+t,"t where 1]t approaches 1] in
this direction, has the least information for estimating t and score function le,rJ' at t == O.
A function X (1]) of the nuisance parameter can, despite the name, also be of interest.
The efficient influence function for this parameter can be found from (25.29). The adjoint
of A e ,,, :}Rk x JHI" r-+ L 2 (P e ,,,), and the corresponding information operator A ,r]Ae,,, : ]Rk x
1HI" r-+ JRk x lin 1HI" are given by, with B;,,,: L 2 (P e ,,, r-+ lin lliI" the adjoint of B e ,,,,
A ,r]g
(Pe, r]gf e , r]' B;, r]g),
( * le''' 'T pe,r] * fe,r]Be,r]' ) ( a b) .
A ,r]Ae,r](a, b) ==
Be,r]fe,r] B(),r]B(),,,
The diagonal elements in the matrix are the information operators for the parameters e and
1], respectively, the former being just the ordinary Fisher information matrix I(),r] for e. If
1] r-+ X (1]) is differentiable as before, then the function (e, 1]) r-+ X (1]) is differentiable with
influence function (O, Xr]). Thus, for a real parameter X (1]), equation (25.29) becomes
7r .€ () - O
(),r] 'p P e . TJ ,r] - ,
B;,r] 1/r Pe,TJ == Xr]'
If I (),r] is invertible and Xr] is contained in the range of B;,r]B(),r]' then the solution lf Pe,TJ of
these equations is
B(),r] (B;,r] B(),r])- Xr] - (B(),,,(B;,r]B(),,,)- Xr]' i(),r]) e l; l(),r]'
,TJ
The second part of this function is the part of the efficient score function for X (1]) that is
"lo st" due to the fact that () is unknown. Because it is orthogonal to the first part, it adds a
positive contribution to the variance.
25.5.2 Information Loss Models
Suppose that a typical observation is distributed as a measurable transformation X == m(Y)
of an unobservable variable Y. Assume that the form of m is known and that the distribution 1]
of Y is known to belong to a class H. This yields a natural parametrization of the distribution
P" of X. A nice property of differentiability in quadratic mean is that it is preserved under
"censoring" mechanisms of this type: If t r-+ 1]t is a differentiable submodel of H, then the
induced submodel t r-+ Pr]t is a differentiable submodel of {Pr] : 1] EH}. Furthermore, the
score function g == Ar]b (at t == O) for the induced model t r-+ Pr], can be obtained from the
score function b (at t == O) of the model t r-+ 1]t by taking a conditional expectation:
A"b(x) == Er] (b(Y) IX == x).
25.5 Score and Informatian Operators
375
If we consider the scores b and g as the carriers of information about t in the variables
Y with law TJt and X with law P TJt , respectively, then the intuitive meaning of the condi-
tional expectation operator is clear. The information contained in the observation X is the
information contained in Y diluted (and reduced) through conditioning. t
25.34 Lemma. Suppose that {TJt : O < t < I} is a collection of probability measures on
a measurable space (Y, B) such that for some measurable function b : Y 1---+ IR
J [ d1]iI2 - d1]1/2 1 ] 2
- -b dr/12 ---+ O.
t 2
For a measurable map m : Y 1---+ X let PTJ be the distribution ofm(Y) if Y has law 1] and let
ATJb(x) be the conditional expectation of b(Y) given m(Y) == x. Then
J [ dP1/2 - dP1/2 1 ] 2
TJt ry _ _ A b dP 1/2 ---+ O.
t 2 TJ TJ
If we consider ATJ as an operator ATJ : L2(1]) 1---+ L2(PTJ)' then its adjoint A : L2(PTJ)
L 2 (1]) is a conditional expectation operator also, reversing the roles of X and Y,
A g(y) ==ETJ(g(X) I Y == y).
This follows because, by the usual rules for conditional expectations, EE(g(X) I Y)b(Y) ==
Eg(X)b(Y) == Eg(X)E(b(Y) IX). In the "calculus of scores" of Theorem 25.3 1 the adjoint
is understood to be the adjoint of Ary : IHITJ L 2 (Pry) and hence to have range lin IHIry C L 2 (1]).
Then the conditional expectation in the preceding display needs to be followed by the
orthogonal projection onto lin IHITJ'
25.35 Example (Mixtures). Suppose that a typical observation X possesses a conditional
density p(x Iz) given an unobservable variable Z == z. If the unobservable Z possesses an
unknown probability distribution 1], then the observations are arandom sample from the
mixture density
Pry(x) = J p(xlz)dYJ(z).
This is a missing data problem if we think of X as a function of the pair Y == (X, Z). A
score for the mixing distribution TJ in the model for Y is a function bez). Thus, a score space
for the mixing distribution in the model for X consists of the functions
Aryb(x) = Ery (b(Z) IX = x) = J bez) p(x IZ) dYJ(z) .
f p(x Iz) d1](z)
If the mixing distribution is completely unknown, which we assume, then the tangent set
iI 1] for TJ can be taken equal to the maximal tangent set {b E L 2 (1]) : 1] b == O} .
In particular, consider the situation that the kern el p(x Iz) belongs to an exponential
family, p(x Iz) == c(z)d (x) exp(zT x). We shall show that the tangent set ATJ ii ry is dense
t For a proof of the following lemma, see, for example, [139, pp. 188-193].
376
Semiparametric Models
in the maximal tangent set {g E L 2 (Pry): Pryg == O}, for every 17 whose support contains
an interval. This has as a consequence that empirical estimators IfD n g, for a fixed squared-
integrable function g, are efficient estimators for the functional1j.J (rJ) == P ry g. For instance,
the sample mean is asymptotically efficient for estimating the mean of the observations.
Thus nonparametric mixtures over an exponential family form very large models, which
are only slightly smaller than the nonparametric model. For estimating a functional such
as the mean of the observations, it is of relatively little use to know that the underlying
distribution is a mixture. More precisely, the additional information does not decrease the
asymptotic variance, although there may be an advantage for finite n. On the other hand,
the mixture structure may express a structure in reality and the mixing distribution 17 may
define the functional of interest.
The closure of the range of the operator Ary is the orthocomplement of the kern el N(A;)
of its adjoint. Rence our claim is proved if this kernel is zero. The equation
o = A g(z) = E(g(X) I Z = z) = ! g(x) p(x Iz) dv(x)
says exactly that g(X) is a zero-estimator under p(x Iz). Because the adjoint is defined
on L 2 (77), the equation 0== A;g should be taken to mean A;g(Z) == O almost surely under
rJ. In other words, the display is valid for every z in a set of rJ-measure 1. If the support
of rJ contains a limit point, then this set is rich enough to conclude that g == O, by the
completeness of the exponential family.
If the support of rJ does not contain a limit point, then the preceding approach fails.
Rowever, we may reach almost the same conclusion by using a different type of scores.
The paths rJt == (1 - ta)rJ + ta771 are well-defined for O < at < 1, for any fixed a > O and
rJ 1, and lead to scores
log p t (x) = a ( Pryl (x) - 1 ) .
3tlt=0 pry
This is certainly a score in a pointwise sense and can be shown to be a score in the L 2 -sense
provided that it is in L2(Pry). If g E L2(Pry) has Pryg == O and is orthogonal to all scores of
this type, then
O=P lg=P g( : -I}
every rJ 1 .
If the set of distributions {Pry : rJ E H} is complete, then we can typically conclude that g == O
almost surely. Then the closed linear span of the tangent set is equal to the nonparametric,
rnaximal tangent set. Because this set of scores is also a convex cone, Theorems 25.20
and 25.21 next show that nonparametric estimators are asymptotically efficient. D
25.36 Example (Semiparametric mixtures). In the preceding example, replace the den-
sity p (x I z) by a parametric family po (x I z). Then the model PO (x I z) d 77 (z) for the un-
observed data Y == (X, Z) has scores for both e and rJ. Suppose that the model t 77t is
differentiable with score b, and that
ff [P a (x Iz) - p /2(x Iz) - aT £0 (x I z) p /2(x Iz) r dfL(X) d1'J(z) = o(lla 112).
25.5 Score and Informatian Operators
377
Then the function aT fe (x Iz) + bez) can be shown to be a score function corresponding to
the model t r-+ Pe+ta(x Iz) d17t(z). Next, by Lemma 25.34, the function
Ee (aT le(X IZ) + beZ) IX =x) = J(le(x Iz) + bez)) pe(x Iz) dry(z)
,rJ f Pe (x Iz) d17(z)
is a score for the model corresponding to observing X only. O
25.37 Example (Random censoring). Suppose that the time T of an event is only ob-
served if the event takes place before a censoring time C that is generated independently of
T; otherwise we observe C. Thus the observation X == (Y, ) is the pair of transformations
Y == T !\ C and == l{T < C} of the "full data" (T, C). If T has a distribution function F
and t r-+ Ft is a differentiable path with score function a, then the submodel t r-+ PPt,G for
X has score function
adF
Ap,Ga(x) == Ep(a(T) IX == (y, 8)) == (1 - 8) (y,oo) + 8a(y).
1 - F (y )
A score operator for the distribution of C can be defined similarly, and takes the form, with
G the distribution of C,
bdG
B p,Gb(x) == (1 - 8)b(y) + 8 [y,oo) .
1 - G_(y)
The scores Ap,Ga and Bp,Gb form orthogonal spaces, as can be checked directly from
the formulas, because EApa(X)BGb(X) == FaGb. (This is also explained by the product
structure in the likelihood. ) A consequence is that knowing G does not help for estimating
F in the sense that the information for estimating parameters of the form 1/1' (P F , G) == X (F)
is the same in the models in which G is known or completely unknown, respectively. To
see this, note first that the influence function of such a parameter must be orthogonal to
every score function for G, because d / dt ljJ (Pp,G) == O. Thus, due to the orthogonality of
the two score spaces, an influence function of this parameter that is contained in the closed
linear span of R(Ap,G) + R(B P,G) is automatically contained in R(Ap,G)' O
25.38 Example (Current status censoring). Suppose that we only observe whether an
event at time T has happened or not at an observation time C. Then we observe the trans-
formation X == (C, l{T < C}) == (C, ) of the pair (C, T). If T and C are independent
with distribution functions F and G, respectively, then the score operators for F and G are
given by, withx == (e,8),
adF adF
A a ( x ) == E ( a ( T ) I C == e == 8 ) == ( 1 - 8 ) (e,oo) + 8 [O,eJ
P,G F , 1 - F(e) F(e)'
Bp,Gb(x) == E(b(C) I C == C, == 8) == b(e).
These score functions can be seen to be orthogonal with the help of Fubini's theorem. If
we take F to be completely unknown, then the set of acan be taken all functions in L 2 (F)
with F a == O, and the adjoint operator A ,G restricted to the set of mean-zero functions in
L 2 (Pp,G) is given by
A ,Gh(c) == 1 h(u, 1) dG(u) + 1 h(u, O) dG(u).
[e,oo) [O,e)
378
Semiparametric Models
For simplicity assume that the true F and G possess continuous Lebesgue densities, which
are positive on their supports. The range of A G consists of functions as in the preceding
,
display for functions h that are contained in L 2 (P F ,G), or equivalently
f h 2 (u, 0)(1 - F)(u) dG(u) < 00 and f h 2 (u, l)F(u) dG(u) < 00.
Thus the functions h(u, 1) and h(u, O) are square-integrable with respect to G on any
interval inside the support of F. Consequently, the range of the adjoint A G contains only
,
absolutely continuous functions, and hence (25.29) fails for every parameter X (F) with an
influence function XF that is discontinuous. More precisely, parameters X (F) with influence
functions that are not almost surely equal under F to an absolutely continuous function.
Because this includes the functions l[o,t] - F(t), the distribution function F f-+ X (F) == F(t)
at a point is not a differentiable functional of the model. In view of Theorem 25.32 this
means that this parameter is not estimable at ,Jn -rate, and the usual normal theory does not
apply to it.
On the other hand, parameters with a smooth influence function XF may be differentiable.
The score operator for the model PF,G is the sum (a, b) f-+ AF,Ga + BF,Gb of the score
operators for F and G separately. Its adjoint is the map h f-+ (A Gh, B; Gh). A parameter
, ,
of the form (F, G) f-+ X (F) has an influence function of the form (XF, O). Thus, for a
parameter of this type equation (25.29) takes the form
A ,G 1f pp,G == XF,
B;,G1f pp,G == O.
The kernel N(A ,G) consists of the functions h E L 2 (P F ,G) such that h(u, O) == h(u, 1)
almost surely under F and G. This is precisely the range of B F, G, and we can conclude that
R(AF,G).l == N(A ,G) == R(BF,G) == N(B;,G).l.
Therefore, we can solve the preceding display by first solving A Gh == XF and next project-
,
ing a solution h onto the closure of the range of A F, G. By the orthogonality of the ranges of
AF,G and BF,G, the latter projection is the identity minus the projection onto R(BF,G). This
is convenient, because the projection onto R(B F,G) is the conditional expectation relative
to C.
For example, consider a function X (F) == F a for some fixed known, continuously dif-
ferentiable function a. Differentiating the equation a == A , G h, we find a' (e) == (h (c, O) -
h (c, 1)) g (c). This can happen for some h E L 2 (P F , G) only if, for any T such that
O < F(T) < 1,
1°°( r(1-F)dG
l'( rFdG
1 00 (h(u, O) - h(u, 1) f(1 - F)(u) dG(u) < 00,
l' (h(u, O) - h(u, 1) f F(u) dG(u) < 00.
If the left sides of these equations are finite, then the parameter P F , G f---+ F a is differentiable.
An influence function is given by the function h defined by
h ( c O ) == a' (e) 1['[,(0) (e)
, g(e)'
and h ( c, 1) = _ a' ( c ) 1 [O. ,) ( c) .
g(e)
25.5 Score and Information Operators
379
The efficient in:fluence function is found by projecting this onto R(Ap,G), and is given by
(h(e, 1) - h(e, O)) (8 - F(e))
1 - F(c) I F(c) I
-8 a (e) + (1 - 8)-a (e).
g(c) g(e)
For example, for the mean X (F) == fud F (u), the influence function certainly exists if the
density g is bounded away from zero on the compact support of F. D
h(e, 8) - Ep,G(h(C, ) I C == e)
*25.5.3 Missing and Coarsening at Random
Suppose that from a given vector (YI, Y 2 ) we sometimes observe only the first coordinate
YI and at other times both YI and Y 2 . Then Y 2 is said to be missing at random (MAR)
if the conditional probability that Y 2 is observed depends only on YI, which is always
observed. We can formalize this definition by introducing an indicator variable that
indicates whether Y 2 is missing ( == O) or observed ( == 1). Then Y 2 is missing at random
if P( == O I Y) is a function of YI only.
If next to P ( == O I Y) we also specify the marginal distribution of Y, then the distribution
of (Y, ) is fixed, and the observed data are the function X == (ep (Y, ), ) defined by (for
instance)
ep(y, O) == YI,
ep (y, 1) == y .
The tangent set for the model for X can be derived from the tangent set for the model
for (Y, ) by taking conditional expectations. If the distribution of (Y, ) is completely
unspecified, then so is the distribution of X, and both tangent spaces are the maximal
"nonparametric tangent space". If we restrict the model by requiring MAR, then the tangent
set for (Y, ) is smaUer than nonparametric. Interestingly, provided that we make no further
restrictions, the tangent set for X remains the nonparametric tangent set.
We shaU show this in somewhat greater generality. Let Y be an arbitrary unobservable
"fuU observation" (not necessarily a vector) and let be an arbitrary random variable.
The distribution of (Y, ) can be deterrnined by specifying a distribution Q for Y and
a conditional density r (8 I y) for the conditional distribution of given Y. t As before,
we observe X == (ep (Y, ), ), but now ep may be an arbitrary measurable map. The
observation X is said to be coarsening at random (CAR) if the conditional densities r (8 I y)
depend on x == (ep (y, 8), 8) only, for every possible value (y, 8). More precisely, r (8 I y) is
a measurable function of x.
25.39 Example (Missing at random). If E {O, I} the requirements are both that
P( == O I Y == y) depends only on ep(y, O) and O and that P( == 11 Y == y) depends only on
ep (y, 1) and 1. Thus the two functions y 1--+ P( == O I Y == y) and y 1--+ P( == 11 Y == y)
may be different (fortunately) but may depend on y only through ep(y, O) and ep(y, 1),
respectively.
If ep (y, 1) == y, then 8 == 1 corresponds to observing y completely. Then the require-
ment reduces to P( == O I Y == y) being a function of ep (y, O) only. If Y == (YI, Y 2 ) and
ep (y, O) == YI, then CAR reduces to MAR as defined in the introduction. D
t The density is relative to a dominating measure von the sample space for L}., and we suppose that (8, y) 1---+ r(8 I y)
is a Markov kern el.
380
Semiparametric Models
Denote by Q and R the parameter spaces for the distribution Q of Y and the kernels
r (8 I y) giving the conditional distribution of given Y, respectively. Let Q x R == (Q x
R: Q E Q, R E R) and P == (PQ,R: Q E Q, R E R) be the models for (Y, ) and X,
respectively.
25.40 Theorem. Suppose that the distribution Q of Y is completely unspecified and the
Markov kernel r(81 y) is restricted by CAR, and only by CAR. Then there exists a tangent set
FPQ,R for the model P == (PQ,R : Q E Q, R E R) who.se closure consists of all mean-zero
functions in L 2 (P Q ,R). Furthermore, any element ofPpQ,R can be orthogonally decompo-
sed as
EQ,R (a(Y) IX == x) + b(x),
. .
where a E QQ and b E RR. The functions a and b range exactly over the functions a E
L 2 (Q) with Qa == O and b E L 2 (P Q ,R) with ER (b(X) I Y) == O almost surely, respectively.
Proof. Fix a differentiable submodel t 1---+ Q t with score a. Furthermore, for every fixed
y fix a differentiable submodel t 1---+ rt (. I y) for the conditional density of given Y == y
with score b a (8 I y) such that
ff [ r 1 /\o I y) r 1 / 2 (o I y) - bo(o I y)r 1 / 2 (o I y)r dv(o)dQ(y) --+ O.
Because the conditional densities satisfy CAR, the function b a (8 I y) must actually be a
function b(x) of x only. Because it corresponds to a score for the conditional model, it
is further restricted by the equations f b a (8 I y) r(8 I y) dv(8) == ER (b(X) I Y == y) == O for
every y. Apart from this and square integrability, b a can be chosen freely, for instance
bounded.
By a standard argument, with Q x R denoting the law of (Y, ) under Q and r,
f [( d Q x R ) 1/2 ( d Q X R ) 1/2 1 ] 2
t t t- - 2(a(y) + b(x») (dQ x R)I/2 --+ O.
Thus a(y) + b(x) is a score function for the model of (Y, ), at Q x R. By Lemma 25.34
its conditional expectation EQ,R (a(Y) + b(X) IX == x) is a score function for the model
of X.
This proves that the functions as given in the theorem arise as scores. To show that
the set of all functions of this type is dense in the nonparametric tangent set, suppose that
some function g E L 2 (P Q ,R) is orthogonal to all functions EQ,R (a(Y) IX == x) + b(x).
Then EQ,Rg(X)a(Y) == EQ,Rg(X)EQ,R (a(Y) IX) == O for all a. Rence g is orthogonal to
all functions of Y and hence is a function of the type b. If it is also orthogonal to all b, then
it must be O. .
The interest of the representation of scores given in the preceding theorem goes beyond
the case that the models Q and R are restricted by CAR only, as is assumed in the theorem.
It shows that, und er CAR, any tangent space for P can be decomposed into two orthogonal
pieces, the first part consisting of the conditional expectations EQ,R (a(Y) I X) of scores a
for the model of Y (and their limit s ) and the second part being scores b for the model R
25.5 Score and Information Operators
381
describing the "missingness pattem." CAR ensures that the latter are functions of x already
and need not be projected, and also that the two sets of scores are orthogonal. By the product
structure of the likelihood q (y) r (8 I y), scores a and b for q and r in the model Q x R are
always orthogonal. This orthogonality may be lost by projecting them on the functions of
x, but not so under CAR, because b is equal to its projection.
In models in which there is apositive probability of observing the complete data, there
is an interesting way to obtain all influence functions of a given parameter PQ,R r--+ X (Q).
Let C be a set of possible values of leading to a complete observation, that is, ep (y, 8) == Y
whenever 8 E C, and suppose that R (C I y) == PR ( E C I Y == y) is positive almost surely.
Suppose for the moment that R is known, so that the tangent space for X consists only of
functions of the form EQ,R(a(Y) IX). If XQ(y) is an influence function of the parameter
Q r--+ x (Q) on the model Q, then
. 1 {8 E C} .
1/1 PQ,R(X) = R(C I y) XQ(Y)
is an influence function for the parameter 1/r(P F ,G) == x (Q) on the model P. To see this,
first note that, indeed, it is a function of x, as the indicator 1 {8 E C} is nonzero only if
(y, 8) == x. Second,
. l{ EC}
E Q ,R1/1P Q ,R(X)E Q ,R(a(Y) IX) = EQ,R R(C I Y) XQ(Y)a(Y)
== EQ,RX'Q(Y) a(Y).
The influence function we have found is just one of many influence functions, the other ones
being obtainable by adding the orthocomplement of the tangent set. This particular influence
function corresponds to ignoring incomplete observations altogether but reweighting the
influence function for the full model to eliminate the bias caused by such neglect. U sually,
ignoring all partial observations does not yield an efficient procedure, and correspondingly
this influence function is usually not the efficient influence function.
All other influence functions, including the efficient influence function, can be found by
adding the orthocomplement of the tangent set. An attractive way of doing this IS:
- by varying XQ over all possible influence functions for Q r--+ x (Q), combined with
- by adding all functions b(x) with ER (b(X) I Y) == O.
This is proved in the following lemma. We still assume that R is known; if it is not, then
the resulting functions need not even be influence functions.
25.41 Lemma. Suppose that the parameter Q r--+ x (Q) on the model Q is differentiable
at Q, and that the conditional probability R(C I Y) == P( E C I Y) of having a complete
observation is bounded away from zero. Then the parameter PQ,R 1---+ X (Q) on the model
(PQ,R: Q E Q) is differentiable at PQ,R and any of its influencefunctions can be written
in the form
1 {8 E C} . b
R(C I y) XQ(Y) + (x),
for X Q an influence function of the parameter Q r--+ x (Q) on the model Q and a function
b E L 2 (P Q ,R) satisfying ER (b(X) I Y) == O. This decomposition is unique. Conversely,
every function of this form is an influence function.
382
Semiparametric Models
Proof. The function in the display with b == O has already been seen to be an influence
function. (N ote that it is square- integrable, as required.) Any function b (X) such that
ER (b(X) I Y) == O satisfies EQ,Rb(X)EQ,R (a(Y) IX) == O and hence is orthogonal to the
tangent set, whence it can be added to any influence function.
To see that the decomposition is unique, it suffices to show that the function as given in
the lemma can be identically zero only if X Q == O and b == O. If it is zero, then its conditional
expectation with respect to Y, which is X Q, is zero, and reinserting this we find that b == O
as well.
Conversely, an arbitrary influence function ljJ PQ.R of PQ,R r-+ X (Q) can be written in the
form
. 1 {<5 E C} . [ . 1 {£5 E C} . ]
1/rp Q ,/X) = R(Cly) X(y)+ 1/rP Q ,R(X)- R(Cly)X(y) ,
For X (Y) == ER ( P (X) I Y ) , the conditional expectation of the part within square brackets
Q.R
with respect to Y is zero and hence this part qualifies as a function b. This function X is an in-
fluence function for Q r-+ x (Q), as follows from the equality EQ,RE R ( PQ.R (X) I Y)a(Y) ==
EQ,R PQ,R (X)EQ,R (a(Y) I X) for every a. .
Even though the functions XQ and b in the decomposition given in the lemma are
uniquely determined, the decomposition is not orthogonal, and (even under CAR) the
decomposition does not agree with the decomposition of the (nonparametric) tangent space
given in Theorem 25.40. The second term is as the functions b in this theorem, but the
leading term is not in the maximal tangent set for Q.
The preceding lemma is valid without assuming CAR. Under CAR it obtains an inter-
esting interpretation, because in that case the functions b range exactl y over all scores for
the parameter r that we would have had if R were completely unknown. If R is known,
then these scores are in the orthocomplement of the tangent set and can be added to any
influence function to find other influence functions.
A second special feature of CAR is that a similar representation becomes available in
the case that R is (p arti ally ) unknown. Because the tangent set for the model (P Q, R : Q E
Q, R E R) contains the tangent set for the model (PQ,R : Q E Q) in which R is known,
the influence functions for the bigger model are a subset of the influence functions of the
smaller model. Because our parameter X (Q) depends on Q only, they are exactly those
influence functions in the smaller model that are orthogonal to the set R PP Q . R of all score
functions for R. This is true in general, also without CAR. Under CAR they can be found
by subtracting the projections onto the set of scores for R.
25.42 Corollary. Suppose that the conditions of the preceding lemma hold and that the
. .
ta gent space PP Q . R for the model (P Q,R : Q E Q, R E R) is taken to. be the sum q PPQ,R +
RPP Q . R of tangent spaces of scores for Q and R sepa rate ly. If QPPQ,R and RPPQ,R are
orthogonal, in particular under CAR, any influencefunction of PQ,R r-+ X (Q)for the model
(PQ,R : Q E Q, R E R) can be obtained b y ta ki'!g the functions given by the preceding
lemma and subtracting their projection onto lin RP PQ,R'
Proof. The influence functions for the bigger model are exactly those influence functions
for the model in which R is known that are orthogonal to R PP Q . R . These do not change
25.5 Score and Informatian Operators
383
by subtracting their projection onto this space. Thus we can find all influence functions as
claimed.
Ifthe score spaces for Q and R are orthogonal, then the projection of an influence function
-- . .
onto lin R PPQ.R is orthogonal to Q PPQ,R' and hence the inner products with elements of this
set are unaffected by subtracting it. Thus we necessarily obtain an influence function. .
The efticient influence function 1[;- p is an influence function and hence can be written
Q,R
in the form of Lemma 25.41 for some XQ and b. By detinition it is the unique influence
function that is contained in the closed linear span of the tangent set. Because the parameter
of interest depends on Q only, the efticient influence function is the same (under CAR or,
. .
more generally, if QP?Q,R 1- RPPQ,R)' whether we assume R known or not. One way of
finding the efficient influence function is to minimize the variance of an arbitrary influence
function as given in Lemma 25.41 over XQ and b.
25.43 Example (Miss ing at random). In the case of MAR models there is a simple rep-
resentation for the functions b(x) in Lemma 25.41. Because MAR is a special case of CAR,
these functions can be obtained by computing all the scores for R in the model for (Y, )
under the assumption that R is completely unknown, by Theorem 25.40. Suppose that
takes only the values O and 1, where 1 indicates a full observation, as in Example 25.39,
and set Jr (y): == P( == 11 Y == y). Under MAR Jr (y) is actually a function of eP (y, O) only.
The likelihood for (f, ) takes the fonn
q (y ) r (tJ I y) == q (y) Jr (y ) 8 (1 - Jr (y ) ) 1-8 .
Insert a path Jrt == Jr + te, and differentiate the log likelihood with respect to t at t == O to
obtain a score for R of the form
tJ 1 - tJ tJ - Jr (y )
-e(y) - c(y) == e(y).
Jr (y ) 1 - Jr (y ) Jr (y) (1 - Jr) (y)
To remain within the model the functions Jr t and Jr, whence c, may depend on y only
through eP (y, O). Apart from this restriction, the preceding display gives a candidate for b
in Lemma 25.41 for any e, and it gives all such b.
Thus, with a slight change of notation any influence function can be written in the form
tJ. tJ - Jr (y)
Jr (y ) X Q (y) - Jr (y) C (y ) .
One approach to finding the efficient influence function in this case is first to minimize the
variance of this influence function with respect to e and next to optimize over X Q' The first
step of this plan can be carried out in general. Minimizing with respect to c is a weighted
least -square s problem, whose solution is given by
c(Y) == EQ,R (XQ (Y) I ep (Y, O)).
To see this it suffices to verify the orthogonality relation, for all c,
tJ. tJ - Jr (y) _ tJ - Jr (y )
Jr (y) X Q (y) - Jr (y ) C (y)..l Jr (y) C (y ) .
Splitting the inner product of the se functions on the first minus sign, we obtain two term s,
both of which reduce to E Q ,RXQ(Y)c(Y)(l - Jr)(Y)jJr(Y). D
384
Semiparametric Models
25.6 Testing
The problem of testing a null hypothesis Ha : 1jf (P) < O versus the alternative HI : 1jf (P) > O
is closely connected to the problem of estimating the function 1jf (P). It ought to be true that
a test based on an asymptotically efficient estimator of 1jf (P) is, in an appropriate sense,
asymptoticallyoptimal. For real-valued parameters 1jf(P) this optimality can be taken in
the absolute sense of an asymptotically (locally) uniformly most powerful test. With higher-
dimensional parameters we run into the same problem of defining a satisfactory notion of
asymptotic optimality as encountered for parametric models in Chapter 15. We leave the
latter case undiscussed and concentrate on real-valued functional s 1jf : P r--+ ffi..
Given a model P and a measure P on the boundary of the hypotheses, that is, 1jf(P) == O,
we want to study the "local asymptotic power" in a neighborhood of P. Defining a local
power function in the present infinite-dimensional case is somewhat awkward, because
there is no natural "rescaling" of the parameter set, such as in the Euclidean case. We
shall utilize submodels corresponding to a tangent set. Given an element g in a tangent
set P p , let t r--+ Pt,g be a differentiable submodel with score function g along which 1jf is
differentiable. For every such g for which -.fr pg == pljf pg > O, the sub model Pt,g belongs
to the alternative hypothesis HI for (at least) every sufficiently small, positive t, because
1jf(Pt,g) == t Pljf pg + o(t) if 1jf(P) == O. We shall study the power at the alternative s Ph/ ,g'
25.44 Theorem. Let the functional1jf : P r--+ JR be differentiable at P relative to the tangent
space P p with efficient influence function 1jf p. Suppose that 1fr(P) == O. Then for every
sequence of power functions P r--+ nn (P) of level-a tests for Ha : 1jf (P) < O, and every
g E P p with pljf pg > O and every h > O,
( Pljf pg )
limsupnn(Ph/ ,g) < 1 - ep Za - h -2 .
n-+oo (P1jf p)I/2
Proof. This theorem is essentially Theorem 15.4 applied to sufficiently rich submodels.
Because the present situation does not fit exactly in the framework of Chapter 15, we
rework the proof. Fix arbitrary hI and gl for which we desire to prove the upper bound.
For notational convenience assume that P gi == 1.
Fix an orthonormal base g p == (gl, . . . , gm) T of an arbitrary finite-dimensional subspace
ofPp (containing the fixed gl). For every g E lin gp, let t r--+ Pt,g be a submodel with score
g along which the parameter 1fr is differentiable. Each of the submodels t r--+ Pt,g is locally
asymptotically normal at t == O by Lemma 25.14. The efore, with sm-I the unit sphere
of JRm
,
(p,7/ ,aTgp : h > O, a E sm-I) 'v'-7 (Nm(ha, I): h > O, a E sm-I),
in the sense of convergence of experiments. Fix a subsequence along which the limsup
in the statement of the theorem is taken for h == hI and g == gl. By contiguity arguments,
we can extract a further subsequence along which the functions nn(Ph/ ,aTg) converge
pointwise to alimit n(h, a) for every (h, a). By Theorem 15.1, the function n(h, a) is the
power function of a test in the normallimit experiment. If it can be shown that this test is
of level a for testing Ha : aTp 1jf p g p == O, then Proposition 15.2 shows that, for every (a, h)
25.6 Testing
385
T -
with a P1jJ pgp > O,
( T - )
a P1jJ pgp
n(h, a) < 1 - <l> Za - h - T - 1/2 '
(P1jJ pgpP1jJ pgp)
The orthogonal projection of 1/; p onto lin gp is equal to (P1/; pg )gp, and has length
P 1/; p g P 1/; p g p . By choosing lin g p large enough, we can ensure that this length is
arbitrarily close to P 1f . Choosing (h, a) == (h 1 , el) completes the proof, because
limsupnn(Phl/J!1,gl) < n(h l , el), by construction.
To complete the pro of, we show that n is of level a. Fix any h > O and an a E sm-l
such that aT P1/; pgp < O. Then
h ( T - )
1jr(P h /.,fii,a T g) = 1jr(P) + y'n a P1jr pgp + 0(1)
is negative for sufficiently large n. Hence P h / J!1,aT g belongs to Ha and
n(h, a) == lim nn (Ph/J!1,aT g) < a.
Thus, the test with power function n is of level a for testing Ha: aTp 1/; p g p < O. By
continuity it is of level ex for testing Ha : aTp 1(; p g p < O. .
As a consequence of the preceding theorem, a test based on an efficient estimator for
1jJ (P) is automatically "locally uniformly most powerful" : Its power function attains the
upper bound given by the theorem. More precisely, suppose that the sequence of estimators
Tn is asymptotically efficient at P and that Sn is a consistent sequence of estimators of its
asymptotic variance. Then the test that rejects Ha : 1jf(P) == O for y'nTn/ Sn > Za attains the
upper bound of the theorem.
25.45 Lemma. Let the functional1jJ : P JR be differentiable at P with 1jJ (P) == O. Sup-
pose that the sequence Tn is regularat P witha N(O, p1(; )-limitdistribution. Furthermore,
2 p -2 .
suppose that Sn --+ Pl/I p' Then, for every h > O and g EPp,
. ( y'nTn ) ( Plfrpg )
hm Ph/J!1 g > Za == 1 - <l> Za - h _ 2 .
n-+oo ' Sn (P1jJ p)1/2
Proof. By the efficiency of Tn and the differentiability of 1jJ, the sequence y'nTn converges
under P h /J!1,g to a normal distribution with mean hP1/; pg and variance P1/; . .
25.46 Example (Wilcoxon test). Suppose that the observations are two independent ran-
dom samples XI, . . . , X n and YI, . . . , Y n from distribution functions F and G, respectively.
To fit this two-sample problem in the present i.i.d. set-up, we pair the two samples and think
of (Xi, Yi) as asingle observation from the product measure F x G on JR2. We wish to
test the nulI hypothesis Ha : f F d G < 4 versus the alternative HI : f F d G > 4. The
Wilcoxon test, which rejects for large values of f IFn dG n , is asymptoticalIy efficient, rel-
ative to the model in which F and G are completely unknown. This gives a different
perspective on this test, which in Chapters 14 and 15 was seen to be asymptoticalIy effi-
cient for testing location in the logistic location-scale family. Actually, this finding is an
386
Semiparametric Models
example of the general principle that, in the situation that the underlying distribution of the
observations is completely unknown, empirical-type statistics are asymptotically efficient
for whatever they naturally estimate or test (also see Example 25.24 and section 25.7). The
present conclusion concerning the Wilcoxon test extends to most other test statistics.
By the preceding lemma, the efficiency of the test follows from the efficiency of the
Wilcoxon statistic as an estimator for the function 1/1' (F x G) == J F d G. This may be
proved by Theorem 25.47, or by the following direct argument.
The model P is the set of all product measures F x G. To generate a tangent set, we can
perturb both F and G. If t 1---+ Ft and t 1---+ G tare differentiable submodels (of the collection
of all probability distributions on IR) with score functions a and b at t == O, respectively, then
the submodel t 1---+ Ft x G t has score function a (x) + b (y). Thus, as a tangent space we may
take the set of all square- integrable functions with mean zero of this type. For simplicity, we
could restrict ourselves to bounded functions a and b and use the paths d Ft == (1 + ta) d F
and d G t == (1 + t b) d G. The closed linear span of the resulting tangent set is the same as
before. Then, by simple algebra,
VrFxG(a, b) = :t 1jr(Ft x Gt)lt=O = f (1 - G_)adF + f FbdG.
We conclude that the function (x, y) 1---+ (1 - G _) (x) + F (y) is an influence function of 1/1' .
This is of the form a (x) + b (y) but does not have mean zero; the efficient influence function
is found by subtracting the mean.
The efficiency of the Wilcoxon statistic is now clear from Lemma 25.23 and the asymp-
totic linearity of the Wilcoxon statistic, which is proved by various methods in Chapters 12,
13, and 20. O
*25.7 Efficiency and the Delta Method
Many estimators can be written as functions cfJ(Tn) of other estimators. By the delta method
asymptotic normality of Tn carries over into the asymptotic normality of ep (T n ), for every
differentiable map ep. Does efficiency of Tn carry over into efficiency of ep (Tn) as well? With
the right definitions, the answer ought to be affirmative. The matter is sufficiently useful
to deserve a discussion and turns out to be nontrivial. Because the result is true for the
functional delta method, applications include the efficiency of the product -limit estimator
in the random censoring model and the sample median in the nonparametric model, among
many others.
If Tn is an estimator of a Euclidean parameter 1/1' (P) and both ep and 1/1' are differentiable,
then the question can be answered by a direct calculation of the normailirnit distributions.
In view of Lemma 25.23, efficiency of Tn can be defined by the asymptotic linearity (25.22).
By the delta method,
(ep (Tn) - ep o 1/1' (P))
ep (p) (Tn -1/1'(P)) + op(l)
1 n
Lep (p) 1// p(X i ) + op(l).
n. 1
l=
The asymptotic efficiency of ep (Tn) follows, provided that the function x 1---+ ep (p) VJ p (x)
is the efficient influence function of the parameter P 1---+ ep o 1/1' (P). If the coordinates of
25.7 Efficiency and the Delta Method
387
1jr p are contained in the closed linear span of the tangent set, then so are the coordinates
of ep (P) 1jr p, because the matrix multiplication by ep (P) means taking linear combinations.
Furthermore, if lj; is differentiable at P (as a statistical parameter on the model P) and ep is
differentiable at 1/1 (P) (in the ordinary sense of calculus), then
ep o lj; ( P t ) - ep o lj; ( p ) ,. ,_
t ep1jl(P) lj; p g == P ep1jl(P) lj; p g.
Thus the function ep (P) 1jr p is an influence function and hence the efficient influence func-
tion.
More involved is the same question, but with Tn an estimator of a parameter in a Banach
space, for instance a distribution in the space D [ - 00, 00] or in a space ,eoo (:F). The question
is empty until we have defined efficiency for this situation. A definition of asymptotic
efficiency of Banach-valued estimators can be based on generalizations of the convolution
and minimax theorems to general Banach spaces. t We shalI avoid this route and take a
more naive approach.
The dual space lIJ)* of a Banach space JI]) is defined as the collection of all continuous,
linear maps d* : llJ) t----* IR. If Tn is a JI])-valued estimator for a parameter lj;(P) E JI]), then d*T n
is a real-valued estimator for the parameter d*lj;(P) E IR. This suggests to defining Tn to
be asymptotically efficient at P E P if -Vn(Tn - lj;(P)) converges under P in distribution
to a tight limit and d*T n is asymptotically efficient at P for estimating d*1/1'(P), for every
d* E JI])*.
This definition presumes that the parameters d* 1/1' are differentiable at P in the sense of
section 25.3. We shalI require a bit more. Say that 1/1' : P JI]) is differentiable at Prelative
. .
to a given tangent set P p ifthere exists a continuous linear map 1/1' p : L 2 (P) t----* JI]) such that,
for every g EPp and a submodel t Pt with score function g,
1/1' ( P t ) - 1/1' ( p ) .
1/1' pg.
t
This implies that every parameter d* 1/1' : P IR is differentiable at P, whence, for every
d* E JI])*, there exists a function 1jr p ,d* : X IR in the closed linear span of rp p such that
d* 1f p (g) == p 1f P ,d* g for every g E rp p. The efficiency of d* Tn for d* 1/1' can next be
understood in terms of asymptotic linearity of d* -Vn(Tn - lj;(P)), as in (25.22), with
influence function lf P,d*'
To avoid measurability issues, we also allow nonmeasurable functions Tn == Tn (X 1, . . . ,
X n ) of the data as estimators in this section. Let both JI]) and lE be Banach spaces.
25.47 Theorem. Suppose that 1/1' : p JI]) is differentiable at P and takes its values in
a subset JI])<p C JI]), nd uppose that ep : JI])<p C JI]) lE is Hadamard-differentiable at 1/1' (P)
tangentially to lin 1/1 p (P p). Then ep o 1/1' : P lE is differentiable at P. If Tn is a sequence
of estimators with values in JI])<p that is asymptotically efficient at P for estimating 1/1' (P),
then ep (T n ) is asymptotically efficient at P for estimating ep o lj; (P).
Proof. The differentiability of ep o lj; is essentially a consequence of the chain rule for
Hadamard-differentiable functions (see Theorem 20.9) and is proved in the same way. The
derivative is the composition ep (P) o p.
t See for example, Chapter 3.11 in [146] for some possibilities and referenees.
388
Semiparametric Models
First, we sho w th t the. limit distribution L of the sequence y'11(Tn -ljJ(P)) concentrates
on the subspace lin ljJ p (Pp). By the Hahn-Banach theorem, for any 5 c TIJ),
lin p (p p ) n 5 == nd*EillJ*: d*t p = o {d E 5: d* d == O}.
For a separable set 5, we can replace the intersection by a countable subintersection. Be-
cause L is tight, it concentrates on a separable set 5, and hence L gives mass 1 to the left
side provided L (d : d* d == O) == 1 for every d* as on the right side. This probability is equal
to N (O, II VI d* p II ) {O} == 1.
Now we can conclude that under the assumptions the sequence y'11(cjJ(Tn) - ep o 1jf(P))
converges in distribution to a tight limit, by the functional delta method, Theorem 20.8.
Furthermore, for every e* E JE*
y'11(e*cjJ(T n ) - e*cjJ o ljJ(P)) == e*ep (p)y'11(Tn -1jf(P)) + op(l),
where, if necessary, we can extend the definition of d* == e*ep (p) to all of II]) in view of
the Hahn-Banach theorem. Because d* E II])*, the asymptotic efficiency of the sequence
Tn implies that the latter sequence is asymptotically linear in the influence function 1/1 p d* .
,
This is also the influence function of the real-valued map e*ep o ljJ, because
e*ep (p) o pg == d*f pg == PVI P,d*g,
g EPp.
Thus, e*cjJ (Tn) IS asymptotically efficient at P for estimating e*cjJ o ljJ(P), for every
e* E JE* . .
The proof of the preceding theorem is relatively simple, because our definition of an
efficient estimator sequence, although not unnatural, is relatively involved.
Consider, for instance, the case that II]) == foo (5) for some set 5. This corresponds to
estimating a (bounded) function s f-+ ljJ(P)(s) by arandom function s f-+ Tn(s). Then the
"marginal estimators" d* Tn include the estimators Trs Tn == Tn (s) for every fixed s - the
coordinate projections Trs : d f-+ d (s) are elements of the dual space foo (5)* -, but include
many other, more complicated functions of Tn as well. Checking the efficiency of every
marginal of the general type d* Tn may be cumbersome.
The deeper result of this section is that this is not necessary. Dnder the conditions
of Theorem 17.14, the limit distribution of the sequence y'11(Tn - ljJ(P)) in fOO(S) is
determined by the limit distributions of these processes evaluated at finite sets of "times"
Sl, . . . , Sk. Thus, we may hope that the asymptotic efficiency of Tn can also be characterized
by the behavior of the marginals Tn (s) only. Our definition of a differentiable parameter
ljJ : P f-+ JI]) is exactly right for this purpose.
25.48 Theorem (Efficiency in ,eOO(S)). Suppose that ljJ : P f-+ fOO(S) is differentiable at
P, and suppose that Tn (s) is asymptotically efficient at P for estimating ljJ (P) (s), for every
s E S. Then Tn is asynlPtotically efficient at P provided that the sequence y'11(Tn -1jf(P))
converges under P in distribution to a tight limit in foo (S).
The theorem is a consequence of a more general principle that obtains the efficiency of
Tn from the efficiency of d* T,1 for a sufficient number of elements d* E ]JJ)*. By definition,
efficiency of T,1 means efficiency of d* 1',1 for all d* E ]JJ)*. In the preceding theorem the
efficiency is deduced from efficiency of the estimators Trs Tn for all coordinate projections Trs
25.7 Efficiency and the Delta Method
389
on £00 (S). The coordinate projections are a fairly small subset of the dual space of £00 (S).
What makes them work is the fact that they are ofnorm 1 and satisfy Ilzlls == sups l1Tszl.
25.49 Lemma. Suppose that 1/1 : p II]) is differentiable at P, and suppose that dl T,7 is
asymptotically efficient at P for estimating dl 1/1 (P) for every dl in a subset II])I C II])* such
that, for SOlne constant C,
Ildll < C sup IdI (d) I.
dl EJI))I , II dl II 1
Then T,7 is asymptotically efficient at P provided that the sequence -Jfi (Tn - 1/1 (P)) is
asymptotically tig ht under P.
Proof The efficiency of all estimators dl Tn for every dl E II])I implies their asymptotic
linearity. This shows that d'Tn is also asymptotically linear and efficient for every dl E
lin II])/. Thus, it is no loss of generality to assume that II])I is a linear space.
By Prohorov' s theorem, every subsequence of -Jfi (Tn -1/1 (P) ) has a further subsequence
that converges weakly under P to a tight limit T. For simplicity, as sume that the whole
sequence converges; otherwise argue along subsequences. By the continuous-mapping the-
orem, d* -Jfi(Tn - 1jf(P)) converges in distribution to d*T for every d* E II])*. By the
assumption of efficiency, the sequence d* -Jfi(Tn - 1/1 (P) ) is asymptotically linear in the
infiuence function lf p d* for every d* E II])/. Thus, the variable d* T is normally distributed
with mean zero and v ance P1f ,d* for every d* E [])', We show below that this is then
automatically true for every d* E II])*.
By Le Cam's third lemma (which by inspection of its proof can be seen to be valid
for general metric spaces), the sequence -Jfi(Tn - 1/1 (P) ) is asymptotically tight under
PII.jn as well, for every differentiable path t Pt. By the differentiability of 1/1, the
sequence -Jfi (Tn - 1jf (PIl -Ji1)) is tight also. Then, exactl y as in the preceding paragraph,
we can conclude that the sequence d* -Jfi (Tn - 1/1 (P1/v!:iiJ) converges in distribution to
a normal distribution with mean zero and variance plf ,d*' for every d* E II])*. Thus,
d* Tn is asymptotically efficient for estimating d*1/I (P) for every d* E II])* and hence Tn is
asymptotically efficient for estimating 1/I(P), by definition.
It remains to prove that a tight, random element T in II]) such that d* T has law N (O,
Ild* P 112) for every d* E IDi necessarily verifies this same relation for every d* E II])*.t First
assume that II]) == £00 (S) and that II])I is the linear space spanned by all coordinate projections.
Because T is tight, there exists a semimetric p on S such that S is totally bounded
and almost all sample paths of Tare contained in UC(S, p) (see Lemma 18.15). Then
automatically the range of 1/1 p is contained in U C (S, p) as well.
To see the latter, we note first that the map s ET (s) T (u) is contained in U C (S, p)
for every fixed u : If p (sm, tm) ---+ O, then T (Sm) - T (tm) ---+ O almost surely and hence in
second mean, in view of the zero-mean normality of T (sm) - T (tm) for every m, whence
lET (sm)T (u) - ET (tm)T (u) I ---+ O by the Cauchy-Schwarz inequality. Thus, the map
s p ( lf P, nu ) (s) == 1T s p ( lf P, n Il) == ( lf P, n Il' lf P, ns ) p == ET (u) T (s)
t The proof of this lemma would be eonsiderably shorter if we knew already that there exists a tight random
element T with values in JI}) such that d*T has a N(O, Ild*-fpll 2)-distribution for every d* E JI})*. Then it
suffiees to show that the distribution of T is uniquely determined by the distributions of d* T for d* E JI})',
390
Semiparametric Models
is contained in the space UC(S, p) for every u. By the linearity and continuity of the
. .
derivative 1/1 p, the same is then true for the map s f---+ 1/1 p (g) (s) for every g in the closed
linear span of the gradients 1(; p 1T as u ranges over S. It is even true for every g in the
. ' u.
tangent set, because 1/1 p (g) (s) == 1/1 p (TIg) (s) for every g and s, and TI the projection onto
the closure of lin 1(; p, 1T u.
By aminor extension of the Riesz representation theorem for the dual space of C (S, p),
the restriction of a fixed d* E JI])* to U C (S, p) takes the form
d*z = ls z(s) d JL (s),
for IL a signed Borel measure on the completion S of S, and z the unique continuous
extension of Z to S. By discretizing IL , using the total boundedness of S, we can construct
a s quence d in lin {Jl's : s E } such that d -+ d* pointwise on UC(S, p). Then
d 1/1 p -+ d* 1/1 p pointwise on P p. Furthermore, d T -+ d* T almost surely, whence in
distribution, so that d* T is normally distributed with mean zero. Because d T - d; T -+ O
almost surely, we also have that
E(d T - d;T)2 == IId p - d; P 1I ,2 -+ O,
. . .
whence d 1/1 p is a Cauchy sequence in L 2 (P). We conclude that d 1/1 p -+ d*l/f p also in
norm and E(d T)2 == Ild p II 2 -+ Ild* p II 2. Thus, d*T is normally distributed with
mean zero and variance Ild* p Ij ,2' '
This concludes the prooffor JI]) equal to lOO(S). A general Banach space JI]) can be embed-
dedinlOO(JI]);), forJI]); == {d' E JI])', Ild'll < I}, by the map d -+ Zd definedasZd(d') ==d'(d).
By assumption, this map is a norm homeomorphism, whence T can be considered to be a
tight random element in lOO(JI]);). Next, the preceding argument applies. .
Another useful application of the lemma concerns the estimation of functional s 1/1 (P) ==
(1/11 (P), 1/12 (P) ) with values in a product JI])I X JI])2 of two Banach spaces. Even though
marginal weak convergence does not imply joint weak convergence, marginal efficiency
implies joint efficiency!
25.50 Theorem (E.fficiency in product spaces). Suppose that 1/1i : P f---+ JI])i is differentiable
at P, and suppose that Tn,i is asymptotically efficient at P for estimating 1/Ii (P),for i == 1,2.
Then (Tn,I, T n ,2) is asymptotically efficient at P for estimating (0/1 (P), 0/2 (P)) provided
that the sequences ../ii (Tn,i - O/i (P)) are asymptotically tight in llJ)i under P, for i == 1, 2.
Proof. Let JI])' be the set of all maps (dl, d 2 ) f---+ d;* (di) for d;* ranging over JI])7 ' and i == 1, 2.
By the Hahn-Banach theorem, lidi II == sup{ Id;* (di) I: Ild;* II == 1, d;* E JI])7}. Thus, the product
norm II (dl, d 2 ) II == lidI II v IId 2 11 satisfies the condition of the preceding lemma (with C == 1
and equality). .
25.51 Example (Random censoring). In section 25.10.1 it is seen that the distribution of
X == (C /\ T, 1 {T < C}) in the random censoring model can be any distribution on the
sample space. It follows by Example 20.16 that the empirical subdistribution functions 1HIOn
and 1HI ln are asymptotically efficient. By Example 20.15 the product limit estimator is a
Hadamard-differentiable functional of the empirical subdistribution functions. Thus, the
product limit -estimator is asymptotically efficient. D
25.8 Efficient Score Equations
391
25.8 Efficient Score Equations
The most important method of estimating the parameter in a parametric model is the method
of maximum likelihood, and it can usually be reduced to solving the score equations
I: 7= 1 fe (Xi) == O, if necessary in a neighborhood of an initial estimate. A natural gener-
alization to estimating the parameter e in a semiparametric model {Pe,rJ : e E e, 17 E H} is
to solve e from the efficient score equations
n
Lle, n (Xi) == O.
i=l
Here we use the efficient score function instead of the ordinary score function, and we
substitute an estimator f7 n for the unknown nuisance parameter. A refinement of this method
has been applied successfully to a number of examples, and the method is likely to work
in many other examples. A disadvantage is that the method requires an explicit form of
the efficient score function, or an efficient algorithm to compute it. Because, in general,
the efficient score function is defined only implicitly as an orthogonal projection, this may
precIude practical implementation.
A variation on this approach is to obtain an estimator fjn (e) of 17 for each given value of
e, and next to solve e from the equation
n
L le' n(e) (Xi) == O.
i=l
If en is a solution, then it is also a solution of the estimating equation in the preceding
display, for fjn == fjn (en)' The asymptotic normality of en can therefore be proved by
the same methods as applying to this estimating equation. Due to our special choice of
estimating function, the nature of the dependence of fj n (e) on e should be irrelevant for the
limiting distribution of ,Jn(e n - e). Informally, this is because the partial derivative of the
estimating equation relative to the e ins ide fj n (e) should converge to zero, as is clear from
our subsequent discussion of the "no-bi as" condition (25.52). The dependence of fjn (e)
on e does play a role for the consistency of en, but we do not discuss this in this chapter,
because the general methods of Chapter 5 apply. For simplicity we adopt the notation as in
the first estimating equation, even though for the construction of en the two-step procedure,
which "profiles out" the nuisance parameter, may be necessary.
In a number of applications the nuisance parameter 17, which is infinite-dimensional,
cannot be estimated within the usual order O(n- 1 / 2 ) for parametric models. Then the
classical approach to derive the asymptotic behavior of Z -estimators - linearization of the
equation in both parameters - is impossible. Instead, we utilize the notion of a Donsker
class, as developed in Chapter 19. The auxiliary estimator for the nuisance parameter should
satisfy t
- ( -1/2 A )
Pen,77lell' 1l == o p n + lien - eli ,
Pe,rJlllen,7]n -le,77112 o, Pen,rJlllen,7]nI12== Op(l).
(25.52)
(25.53)
t The notation P£fj is an abbreviation for the integral f £fj(x) dP(x). Thus the expeetation is taken with respeet
to x only and not with respeet to fj.
392
Semiparametric Models
The second condition (25.53) merel y requires that the "plug-in" estimator le, fJn is a consistent
estimator for the true efficient influence function. Because Pen,rylen,r/ == O, the first condition
(25.52) requires that the "bias" of the plug-in estimator, due to estimating the nuisance
parameter, converge to zero faster than 1/,Jn. Such a condition comes out naturally of the
proofs. A p arti al motivation is that the efficient score function is orthogonal to the score
functions for the nuisance parameter, so that its expectation should be insensitive to changes
In ry.
25.54 Theorem. Suppose that the model {Pe,ry : e E EJ} is differentiable in quadratic mean
with respect to e at (e, 1'}) and let the efficient information matrix le, ry be nonsingular. Assume
that (25.52) and (25.53) hold. Let en satisfy ,JnIP n l el1 ,fJ/1 ==op(l) and be consistentfor
e. Furthermore, suppose that there exists a Donsker class with square-integrable envelope
function that contains every function le l1 ,fJn with probability tending to 1. Then the sequence
e n is asymptotically efficient at (e, ry).
Proof. Let G n (ef, ryf) == ,Jn (IP n - P e , ry )le', ry' be the empirical process indexed by the func-
tions le',ry" By the assumption that the functions le,fJ are contained in a Donsker class,
together with (25.53),
Gn(e n , n) == Gn(e, 1'}) + op(l).
(see Lemma 19.24.) By the defining relationship of en and the "no-bias" condition (25.52),
this is equivalent to
-Jn(Pe n ,1} - Pe,ry)len,fJn == Gn(e, ry) + op(l + -Jnlle n - eoll).
The remainder of the proof consists of showing that the left side is asymptotically equivalent
to (le,ry +op(l) ),Jn(e n -e), from which the theoremfollows. Because le,ry == Pe,ryle,ryi ,1}'
the difference of the left side of the preceding display and le, 1},Jn (e n - e) can be written
as the sum of three terms:
c f - ( 1/2 1/2 ) [( 1/2 1/2 ) 1" T . 1/2 ]
V n fen,fJn Pen,ry + Pe,ry Pen,ry - Pe,ry - 2(e n - e) f e ,1] Pe,1] dfL
f - ( 1/2 1/2 ) l'T 1/2 C "
+ fe 1" 7 Pt5 -PLJ n -fenPendJLvn(en-e)
n, n 0/1,1] 0,,/ 2 "/ "I
- f (le"'!i,, -le, ) i , Pe, dJ-L -Jnce n - e),
The first and third term can easily be seen to be o p (,Jnlle n - e II) by applying the Cauchy-
Schwarz inequality together with the differentiability of the model and (25.53). The square
of the norm of the integral in the middle term can for every sequence of constants m n 00
be bounded by a multiple of
m f Ille",ry" II P : I p - P : I d J-L 2
+ f Ille,,,ryJ 2 CPe,,, + Pe, ) d J-L [ II ie, 11 2 Pe, dJ-L.
J ll f e "711 >m n
In view of (25.53), the differentiability of the model in e, and the Cauchy-Schwarz inequal-
ity, the first term converges to zero in probability provided m n 00 sufficiently slowly
25.8 Efficient Score Equations
393
to ensure that m n lien - 811 O. (Such a sequence exists. If Zn O, then there exists
asequence£n OsuchthatP(IZnl > 8n) O. Then8;;1/2 Zn O.) Inviewofthelast
part of (25.53), the second term converges to zero in probability for every m n 00. This
concludes the proof of the theorem. .
The preceding theorem is best understood as applying to the efficient score functions
£e,1J' However, its proof only uses this to ensure that, at the true value (e, 17),
- -'T
I e,1J == Pe,17£e,1J£e1J'
The theorem remain s true for arbitrary, mean-zero functions le, 1J provided that this identity
holds. Thus, if an estimator (e, ) only approximately satisfies the efficient score equation,
then the latter can be replaced by an approximation.
The theorem applies to many examples, but its conditions may be too stringent. A
modification that can be theoretically carried through under minimal conditions is based
on the one-step method. Suppose that we are given a sequence of initial estimators en that
is .Jfl-consistent for e. We can assume without loss of generality that the estimators are
discretized on a grid of meshwidth n -1/2, which simplifies the constructions and proof.
Then the one-step estimator is defined as
e n ==e n + ( '0 le ;; Ir ;;(Xi) ) -l'0 le ;;(Xi).
n"ln,1 n"ln,1 ""111,1
i=1 i=1
The estimator en can be considered a one-step iteration of the Newton-Raphson algorithm
for solving the equation L le,fj(X i ) == O with respect to e, starting at the initial guess en.
For the benefit of the simple pro of, we have made the estimators n,i for 17 dependent on
the index i. In fact, we shall use only two different values for n,i' one for the first half of
the sample and another for the second half. Given estimators n == fin (X 1, . . . , X n ) define
n,i by, with m == Ln/2J,
'" { m (X 1, . . . , X m )
17n i == '"
, 17n-m (X m + 1 , . . . , X n )
ifi>m
if i < m.
Thus, for Xi belonging to the first half of the sample, we use an estimator n,i based on
the second half of the sample, and vice versa. This sample-splitting trick is convenient
in the proof, because the estimator of 17 used in le,1J (Xi) is always independent of Xi,
simu1taneously for Xi running through each of the two halves of the sample.
The discretization of en and the sample-splitting are mathematical devices that rarely are
useful in practice. However, the conditions of the preceding theorem can now be relaxed
to, for every deterministic sequence en == e + O(n- 1 / 2 ),
r:: - p
v nP e ",1}£e/1,fjll O,
p eli , 1} lli e/1 , fj /1 - le", 1} 11 2 O.
(25.55)
f 11 - 1/2 - 1/2 11 2
£ e/1 . 1J d P e/1 , 1J - £ e , 1J d P e , 1} O.
(25.56)
25.57 Theorem. Suppose that the model {P e ,1} : e E e} is differentiable in quadratic
mean with respect to e at (8, 17), and let the efficient information matrix le, 1} be nonsingular.
394
Semiparametric Models
Assume that (25.55) and (25.56) hold. Then the sequence en is asymptotically efficient at
(e,17).
Proof. Fix a deterministic sequence of vectors en == e + O (n -1/2). By the sample-splitting,
the first half of the sum L len' n.i (Xi) is a sum of conditionally independent terms, given
the second half of the sample. Thus,
Ee", ( ,JmJP> m (le",1)",; -le", ) I X m + 1 , . . . , X n ) ,Jm Pe", le",1)";'
vare", ( ,JmJP> m (le",1)",; -le", ) I X m + 1 , .. . , Xn) < Pe", Il l e",1)"" -le", 11 2 .
Both expressions converge to zero in probability by assumption (25.55). We conclude
that the sum inside the conditional expectations converges conditionally, and hence also
unconditionally, to zero in probability. By symmetry, the same is true for the second half
of the sample, whence
-JnII»n(le/l' n,i -len,rJ) O.
We have proved this for the probability under (en, 17), but by contiguity the convergence is
also under (e, 17).
The second part of the pro of is technical, and we only report the result. The condition
of differentiabily of the model and (25.56) imply that
-JnII»n (len,rJ -le,rJ) + -JnPe'rJle'rJi 'rJ(en - e) O
(see [139], p. 185). Under stronger regularity conditions, this can also be proved by a Taylor
expansion of le, rJ in e.) By the definition of the efficient score function as an orthogonal
projection, Pe'rJle'rJi 'rJ == 1 e,rJ' Combining the preceding displays, we find that
-JnII»n (len' n,i -le,rJ) + 1 e,rJ-Jn(e n - e) o.
In view of the discretized nature of en, this remain s true if the deterministic sequence en is
replaced by en; see the argument in the proof of Theorem 5.48.
N ext we study the estimator for the information matrix. For any vector h E JR.k, the
triangle inequality yields
2
II» ( h TlA ) 2 - J JP> ( h T l ) 2 < II» ( h TlA - h T l ) 2
m en,rJn,i m en,rJ - m en,rJn,i en,rJ'
By (25.55), the conditional expectation under (en, 17) of the right side given X m + 1 , . . . , X n
converges in probability to zero. A similar statement is valid for the second half of the
observations. Combining this with (25.56) and the law of large numbers, we see that
- -T p-
II» n '€e n , n.i '€e n , Il,i --+ le, rJ .
In view of the discretized nature of en, this remain s true if the deterministic sequence en is
replaced by en.
25.8 Efficient Score Equations
395
The theorem follows combining the results of the last two paragraphs with the definition
of en. .
A further refinement is not to restrict the estimator for the efficient score function to
be a plug-in type estimator. Both theorems go through if le,fj is replaced by a general
estimator ln,e ==£n,e('1 X 1 ,..., X n ), provided that this satisfies the appropriately modi-
fied conditions of the theorems, and in the second theorem we use the sample-splitting
scheme. In the generalization of Theorem 25.57, condition (25.55) must be replaced
by
A P
V nPen,TJln,en ---+ O,
Pen, TJ Illn,e n -len, TJ 11 2 O.
(25.58)
The proofs are the same. This opens the door to more tricks and further relaxation of
the regularity conditions. An intermediate theorem concerning one-step estimators, but
without discretization or sample-splitting, can also be proved under the conditions of The-
orem 25.54. This removes the conditions of existence and consistency of solutions to the
efficient score equation.
The theorems reduce the problem of efficient estimation of e to estimation of the efficient
score function. The estirnator of the efficient score function must satisfy a "no-bias" and a
consistency conditions. The consistency is usually easy to arrange, but the no-bias condition,
such as (25.52) or the first part of (25.58), is connected to the structure and the size of the
model, as the bias of the efficient score equations must converge to zero at a rate faster than
1/,jii. Within the context of Theorem 25.54 condition (25.52) is necessary. If it fails, then
the sequence en is not asymptotically efficient and may even converge at a slower rate than
,jii. This follows by inspection of the proof, which reveals the following adaptation of the
theorem. We assume that le, TJ is the efficient score function for the true parameter (e, 'f})
but allow it to be arbitrary (mean-zero) for other parameters.
25.59 Theorem. Suppose that the conditions ofTheorem 25.54 hold except possibly con-
dition (25.52). Then
r:: " 1 --1 - -
.yn(e n -e)= ,jii {;;/(}'".€(}''' (X;) +.ynPĐn,,,.€Đn' n +op(1).
Because by Lemma 25.23 the sequence en can be asymptotically efficient (regular with
N (O, j; )-limit distribution) only if it is asymptotically equivalent to the sum on the right,
condition (25.52) is seen to be necessary for efficiency.
The verification of the no-bias condition may be easy due to special properties of the
model but may also require considerable effort. The derivative of Pe,TJle,fj with respect
to () ought to converge to a / ae Pe,TJle,TJ == O. Therefore, condition (25.52) can usually be
simplified to
- p
v nPe,TJle,fjn ---+ O.
The dependence on fj is more interesting and complicated. The verification may boil down to
a type of Taylor expansion of Pe,TJle,fj in fj combined with establishing a rate of convergence
for fj. Because 17 is infinite-dimensional, a Taylor series may be nontrivial. If i7 - 'f} can
396
Semiparametric Models
occur as a direction of approach to 17 that leads to a score function B e , 17 ( - 17), then we can
write
P e ,17 fe ,fj
(P e ,17 - Pe,fj)(le,fj -le,17)
_ l [ pe,fj - Pe,17 - B ( /'. _ )]
e,17 e,17 e,17 17 17 .
Pe,17
(25.60)
We have used the fact that P e ,17 le ,17 Be,17 h == O for every h, by the orthogonality property of
the efficient score function. (The use of B e , 17 ( - 17) corresponds to a score operator that
yields scores B e , 17 h from paths of the form 17t == rJ + t h. If we use paths d rJt == (1 + t h) d 17,
then Be,17(d /d17 - 1) is appropriate.) The display suggests that the no-bias condition
(25.52) is certainly satisfied if II - rJ II == O p (n- 1/2 ), for II . II a norm relative to which the
two term s on the right are both of the order o p (II - 1711). In cases in which the nuisance
parameter is not estimable at Jfl-rate the Taylor expansion must be carried into its second-
order term. If the two terms on the right are both O p (II - 1711 2 ), then it is stiH sufficient
to have II - 1711 == o p (n -1/4). This observation is based on a crude bound on the bias, an
integral in which cancellation could occur, by norms and can therefore be too pessimistic
(See [35] for an example.) Special properties of the model may also allow one to take
the Taylor expansion even further, with the lower order derivatives vanishing, and then a
slower rate of convergence of the nuisance parameter may be sufficient, but no examples
of this appear to be known. However, the extreme case that the expression in (25.52)
is identically zero occurs in the important class of models that are convex-linear in the
parameter.
25.61 Example (Convex-linear models). Suppose that for every fixed e the model
{P e ,17 : rJ E H} is convex-linear: H is a convex subset of a linear space, and the depen-
dence rJ 1---+ P e , 17 is line ar. Then for every pair (rJ 1, rJ) and number O < t < 1, the convex
combination rJt == t rJ 1 + (1- t) rJ is a parameter and the distribution t P e , 171 + (1- t) P e , 17 == P e , 17t
belongs to the model. The score function at t == O of the submodel t 1---+ Pe, 17t is
dP e ,171
log d Pe ,t171 +(1-t)17 == - 1.
atlt=o dP e ,17
a
Because the efficient score function for e is orthogonal to the tangent set for the nuisance
parameter, it should satisfy
...., ( dPe'171 ) ....,
0== Pe,17fe,17 d - 1 == P e ,171 fe,rJ'
e,17
This means that the unbiasedness conditions in (25.52) and (25.55) are triviaHy satisfied,
with the expectations Pe,17le,fj even equal to O.
A particular case in which this convex structure arises is the case of estimating a
linear functional in an information-Ioss model. Suppose we observe X == m(Y) for a
known function m and an unobservable variable Y that has an unknown distribution rJ
on a measurable space (Y, A). The distribution P 17 == 17 o m -1 of X depends linearly on
17 . Furthermore, if we are interested in a linear function e == X (17 ), then the nuisance-
parameter space He == { 17 : X (rJ) == e } is a convex subset of the set of probability measures on
(Y, A). O
25.8 Efficient Score Equations
397
25.8.1 Symmetric Location
Suppose that we observe arandom sample from a density TJ(x - 8) that is symmetric about
8. In Example 25.27 it was seen that the efficient score function for 8 is the ordinary score
function,
- TJ'
£e,17(x) == --ex - 8).
TJ
We can apply Theorem 25.57 to construct an asymptotically efficient estimator sequence for
8 under the minimal condition that the density TJ has finite Fisher information for location.
First, as an initial estimator en, we may use a discretized Z -estimator, solving ]p n 1/1 (x -
8) == O for a well-behaved, symmetric function 1/1. For instance, the score function of the
logistic density. The vn-consistency can be established by Theorem 5.21.
Second, it suffices to construct estimators fn,e that satisfy (25.58). By symmetry, the
variables Ti == IX i - 81 are, for a fixed 8, sampled from the density g(s) == 2TJ(s)l{s > O}.
We use these variables to construct an estimator kn for the function g' / g, and next we set
fn,e(x; XI,..., X n ) == -kn (Ix - 81; TI,..., Tn) sign(x - 8).
Because this function is skew-symmetric about the point 8, the bias condition in (25.58) is
satisfied, with a bias of zero. Because the efficient score function can be written in the form
- g'
£e,17 (x) == - - (Ix - 8 I) sign(x - 8),
g
the consistency condition in (25.58) reduces to consistency of kn for the function g' / g in
that
f (kn - )\S)g(S)dS o.
(25.62)
Estimators kn can be constructed by several methods, a simple one being the kernel method
of density estimation. For a fixed twice continuously differentiable probability density w
with compact support, a bandwidth parameter (5 n, and further positive tuning parameters
an, f3n, and Yn, set
1 n ( S - 1',. )
gn(s) == - LW l ,
(5n i=I (5n
AI
A gn
kn (s) == -;-(s)I Sn (s),
gn
Bn == {s: Ig (s)1 < an, gn(s) > f3n, S > Yn}'
(25.63)
Then (25.58) is satisfied providedan t 00, f3n -t O, Yn -t O, and (5n -t O at appropriate speeds.
The proof is technical and is given in the next lemma.
This particular construction show s that efficient estimators for 8 exist under minimal
conditions. It is not necessarily recommended for use in practice. However, any good initial
estimator en and any method of density or curve estimation may be substituted and will
lead to a reasonable estimator for 8, which is theoretically efficient under some regularity
conditions.
398
Semiparametric Models
25.64 Lemma. Let TI, . . . , Tn be arandom sample from a density g that is supported and
absolutely continuous on [O, (0) and satisfies J (g' / ,Jg)2 (s) ds < 00. Then kn given by
(25.63) for a probability density (j) that is twice continuously differentiable and supported
on [-1, 1] satisfies (25.62), if an t 00, Yn + O, fJn + O, and an + O in such away that
an < Yn, a an/ f3 --+ O, na: f3 --+ 00.
Proof. Start by noting that Ilglloo < Jlg'(s)1 ds < jT;, by the Cauchy-Schwarz inequal-
ity. The expectations and variances of g n and its derivative are given by
gn(s) := Egn(s) = E > u c TI ) = f g(s - ay) w(y) dy,
vargn(s) = --;'varw ( s - TI ) < --;'lIwll ,
na a na
Eg (s)=g (s)= fg'(s-aY)W(y)d Y , (s > y),
varg (s) < IIw'II .
na
B Y the dominated -convergence theorem, g n (s) --+ g (s), for every s > O. Combining this
with the preceding display, we conclude that g n (s) g (s). If g' is sufficiently smooth,
then the analogous statement is true for g (s). Under only the condition of finite Fisher
information for location, this may fail, but we stiH have that g (s) - g (s) O for every
s; furthermore, g 1 [0',(0) --+ g' in LI, because
1 00 Ig - g'l(s)ds < f flg'(s - ay) - g'(s) I dsw(y)dy -+ o,
by the LI-continuity theorem on the inner integral, and next the dominated-convergence
theorem on the outer integral.
The expectation of the integral in (25.62) restricted to the complement of the set E n is
equal to
f ( )\S)g(S)P(lg I(S) > cl ar gn(s) < f3 ars < y)ds.
This converges to zero by the dominated-convergence theorem. To see this, note first
that p(gn (s) < fJ) converges to zero for all s such that g(s) > O. Second, the probability
p(lg l(s) > a) is bounded above by 1{lg l(s) > a/2} + 0(1), and the Lebesgue measure
of the set {s : I g I (s) > a /2} converges to zero, because g --+ g' in LI.
On the set En the integrand in (25.62) is the square of the function (g /gn - g'/g)gI/2.
This function can be decomposed as
AI ( AI I ) 1/2 , ( A ) ( I I )
gn ( 1/2 _ 1/2 ) + gn - gn gn _ gn gn - gn + _
A g g n A 1/2 A 1/2 1/2'
gn gn gn gn gn g
On E n the sum of the squares of the four terms on the right is bounded above by
2 1 1 ( I ) 2 ( ' I ) 2
a A' I 2 gn A 2 gn g
f32 lgn - gl + f32 (gn - gn) gn + f32 g /2 (gn - gn) + g /2 - gl/2 .
25.8 Efficžent Score Equations
399
The expectations of the integrals over B n of these four term s converge to zero. First, the
integral over the first term is bounded above by
a f [ 19 (s - (ft) - g(s)1 w(t)dtds < a2 f lg/(t)1 dt f Itlw(t)dt.
}s>y
N ext, the sum of the second and third terms gives the contribution
1 1 2 f 1 2 f ( g ) 2
4 2 11w 1100 gn(s)ds + 2 2 11wll00 172 ds.
na {3 na gn
The first term in this last display converges to zero, and the second as well, provided the
integral remains finite. The latter is certainly the case if the fourth term converges to zero.
By the Cauchy-Schwarz inequality,
(f g/(S - ay) w(y) dy)2 f ( gl ) 2
f ( ) ()d < / 2 (s-ay)w(y)dy.
g s - ay w y y g
U sing Fubini' s theorem, we see that, for any set B, and B a its a -enlargement,
i ( g 2 r (s) ds < ia ( g ;2 r ds.
In particular, we have this for B == B a == IR, and B == {s : g (s) == O}. For the second
choice of B, the sets B a decrease to B, by the continuity of g. On the complement of B,
g / g /2 --+ gl / gl/2 in Lebesgue measure. Thus, by Proposition 2.29, the integral of the
fourth term converges to zero. .
25.8.2 Errors-in- Variables
Let the observations be arandom sample of pairs (Xi, Yi) with the same distribution as
X==Z+e
Y == ex + Z + f,
for a bivariate normal vector (e, f) with mean zero and covariance matrix 2: and arandom
variable Z with distribution 1], independent of (e, f). Thus Y is a linear regression on a
variable Z which is observed with error. The parameter of interest is e == (ex, {3, 2:) and
the nuisance parameter is 1]. To make the parameters identifiable one can put restrictions
on either 2: or fJ. It suffices that fJ is not normal (if adegenerate distribution is considered
normal with variance zero); alternatively it can be assumed that 2: is known up to a scalar.
Given (e, 2:) thestatistic 1/1e(X, Y) == (1, )2:-1(X, Y -ex)T is sufficient(andcomplete)
for 17. This suggests to define estimators for (ex, f3, 2:) as the solution of the "conditional
score equation" ]p n le, 1] == O, for
le,rJ(X' Y) == fe,rJ(X, Y) - Ee(fe,rJ(X, Y) l1/1e(X, Y)).
This estimating equation has the attractive property of being unbiased in the nuisance
parameter, in that
Pe,Tj£e,rJ' == O,
every e, TJ, 1]1.
400
Semiparametric Models
Therefore, the no-bias condition is trivially satisfied, and the estimator need only be
consistent for 17 (in the sense of (25.53)). One possibility for is the maximum likelihood
estimator, which can be shown to be consistent by Wald' s theorem, under some regularity
conditions.
As the notation suggests, the function le, 17 is equal to the efficient score function for
e. We can prove this by showing that the closed linear span of the set of nuisance scores
contains all measurable, square-integrable functions of 1/1e (x, y), because then projecting
on the nuisance scores is identical to taking the conditional expectation.
As explained in Example 25.61, the functions Pe,171 I Pe,17 - 1 are score functions for the
nuisance parameter (at (e, 17)). As is clear from the factorization theorem or direct calcu-
lation, they are functions of the sufficient statistic 1/1e (X, Y). If some function b ( 1/1e (x, y) )
is orthogonal to all scores of this type and has mean zero, then
Ee, 17] b ( 1/1 e (X, Y)) == Ee, 17 b ( 1/1 e (X, Y)) ( Pe, 17] - 1 ) == O.
Pe,l]
Consequent1y, b == O almost surely by the completeness of 1/1e(X, Y).
The regularity conditions of Theorem 25.54 can be shown to be satisfied under the
condition that f Izl 9 d17(z) < 00. Because all coordinates of the conditional score function
can be written in the form Qe (x, y) + Pe (x, Y )E 17 (z l1/1e (X, Y)) for polynomials Qe and
Pe of orders 2 and 1, respectively, the following lemma is the main part of the verification. t
25.65 Lemma. For every O < a < 1 and every probability distribution 170 on JR and
compact K C (O, (0), there exists an open neighborhood U of 170 in the weak topology
such that the class :F of all functions
f z e z (b o +b 1 x+b 2 y) e- cz2 d 17 (z)
(x, Y) 1-+ (aa + al x + a2Y) J ez(bo+hx+b2Y) e-cz2 d 11 (z) ,
with 17 ranging over U, c ranging over K, and a and b ranging over compacta in m:?,
satisfies
( ) V
1 5+2a+4 V+8 Vj2
logN[](t:,:F, L 2 (P)) < C (P(l + Ixl + IYI) /),
for every V > Ila, every measure P on }R2 and 8 > O, and a constant C depending only
on a, 170, U, V, the compacta, and 8.
25.9 General Estimating Equations
Taking the efficient score equation as the basis for estimating a parameter is motivated by
our wish to construct asyrnptotically efficient estimators. Perhaps, in certain situations, this
is too rnuch to ask, and it is better to airn at estimators that corne close to attaining efficiency
or are efficient only at the elements of a certain "ideal submodel." The pay off could be a
gain in robustness, finite-sample properties, or computational simplicity. The information
bounds then have the purpose of quantifying how much efficiency has possibly been lost.
t See [108] for a proof.
25.9 General Estimating Equations
401
We retain the requirement that the estimator is ,jn-consistent and regular at every dis-
tribution P in the model. A somewhat stronger but still reasonable requirement is that it be
asymptotically linear in that
1 n .
(Tn -1/I(P)) == ,jn L1/I P(Xi) + op(l).
n. 1
l=
This type of expansion and regularity implies that 1/1 p is an influence function of the
parameter 1jJ(P), and the difference p -{fp must be orthogonal to the tangent set P p .
This suggests that we compute the set of all influence functions to obtain an indication
of which estimators Tn might be possible. If there is a nice parametrization 1/1 f), T of these
sets of functions in terms of a parameter of interest () and a nuisance parameter i, then a
possible estimation procedure is to solve () from the estimating equation, for given i,
n
L f)'T(Xi) == O.
i=l
The choice of the parameter i determines the efficiency of the estimator e. Rather than
fixing it at same value we also can make it data-dependent to obtain efficiency at every
element of a given submodel, or perhaps even the whole model. The resulting estimator
can be analyzed with the help of, for example, Theorem 5.31.
If the model is parametrized by a partitioned parameter ((), 1]), then any influence function
for e must be orthogonal to the scores for the nuisance parameter 1]. The parameter i might
be indexing both the nuisance parameter 1] and "position" in the tangent set at a given
((), 1]). Then the unknown 17 (or the aspect of it that plays a role in i) must be replaced by an
estimator. The same reasoning as for the "no-bias" condition discussed in (25.60) allows
us to hope that the resulting estimator for () behaves as if the true 1] had been used.
25.66 Example (Regression). In the regression model considered in Example 25.28, the
set of nuisance scores is the orthocomplement of the set eH of all functions of the form
(x, y) 1-* (y - gf)(x))h(x), up to centering at mean zero. The efficient score function for
() is equal to the projection of the score for () onto the set eH, and an arbitrary influence
function is obtained, up to a constant, by adding any element from eH to this. The estimating
equation
n
L (Yi - g f) (X i) ) h (X i) == O
i=l
leads to an estimator with influence function in the direction of (y - gf)(x) )h(x). Because
the equation is unbiased for any h, we easily obtain ,jn-consistent estimators, even for data-
dependent h. The estimator is more efficient if h is closer to the function gf) (x) /E17 (e 2 I X ==
x), which gives the efficient influence function. For full efficiency it is necessary to estimate
the function x 1----+ Er] (e 2 I X == x) nonparametrically, where consistency (for the right norm)
suffices. O
25.67 Example (Missing at random). In Lemma 25.41 and Example 25.43 the influence
functions in a MAR model are characterized as the sums of reweighted influence functions
in the original model and the influence functions obtained from the MAR specification. If
402
Semiparametric Models
the function Jr is known, then this leads to estimating equations of the form
n f1 i . n f1 i - Jr ( Yi )
" 'I Ir ( X. ) - " C ( y. ) - O
-8 Jr(Y i ) 'p e,T l -8 Jr(Y i ) l - .
For instance, if the original model is the regres sion model in the preceding example, then
e,T(Y) is (y - ge(x) )h(x). The efficiency of the estimator is influenc.ed by the choice of
c (the optimal choice is given in Example 25.43) and the choice of 1jf e, T' (The efficient
influence function of the original model ne ed not be efficient here.) If Jr is correctly
specified, then the second part of the estimating equation is unbiased for any c, and the
asymptotic variance when using arandom c should be the same as when using the limiting
value of c. O
25.10 Maximum Likelihood Estimators
Estimators for parameters in semiparametric models can be constructed by any method -
for instance, M -estimation or Z-estimation. However, the most important method to obtain
asymptotically efficient estimators may be the method of maximum likelihood, just as in
the case of parametric models. In this section we discuss the definitions of likelihoods and
give some examples in which maximum likelihood estimators can be analyzed by direct
methods. In Sections 25.11 and 25.12 we discuss two general approaches for analyzing
these estimators.
Because many semiparametric models are not dominated ar are defined in terms of
densities that maximize to intinity, the functions that are called the "likelihoods" of the
models must be chosen with care. For some models a likelihood can be taken equal to a
density with respect to a dominating measure, but for other model s we use an "empirical
likelihood." Mixtures of these situations occur as well, and sometimes it is fruitful to incor-
porate a "penalty" in the likelihood, yielding a "penalized likelihood estimator" ; maximize
the likelihood over a set of parameters that changes with n, yielding a "sieved likelihood
estimator"; or group the data in some way before writing down a likelihood. To bring out
this difference with the classical, parametric maximum likelihood estimators, our present
estimators are sometimes referred to as "nonparametric maximum likelihood estimators"
(NPMLE), although semiparametric rather than nonparametric seems more correct. Thus
we do not give an abstract detinition of "likelihood," but describe "likelihoods that work"
for particular examples. We denote the likelihood for the parameter P given one observation
x by lik(P)(x).
Given a measure P, write P {x} for the measure of the one-point set {x}. The function
x r-+ P {x} may be considered the density of P, or its absolutely continuous part, with respect
to counting measure. The empiricallike lihood of a sample XI, . . . , X n is the function,
n
p r-+ Il P{X i }.
i=l
Given a model P, a maximum likelihood estimator could be defined as the distribution P
that maximizes the empiricallikelihood over P. Such an estimator may or may not exist.
25.68 Example (Empirical distribution). Let P be the set of all probability distributions
on the measurable space (X, A) (in which one-point sets are measurable). Then, for n
25.10 Maximum Likelihood Estimatars
403
fixed different values XI, .. . , X n , the vec tor (P {XI}, . .., P{Xn}) ranges over all vectors
p > O such that L Pi < 1 when P ranges over P. To maximize P Ili Pi, it is
clearly best to choose P maximal: Li Pi == 1. Then, by symmetry, the maximizer must
be P == (1/ n, . . . , 1/ n). Thus, the empirical distribution P n == n-I L eS Xi maximizes
the empirical likelihood over the nonparametric model, whence it is referred to as the
nonparametric maximum likelihood estimator.
If there are ties in the observations, this argument must be adapted, but the result is the
same.
The empiricallikelihood is appropriate for the nonparametric model. For instance, in the
case of a Euclidean space, even if the model is restricted to distributions with a continuous
Lebesgue density p, we still cannot use the map P Il7=IP(X i ) as a likelihood. The
supremum of this likelihood is infinite, for we could choose P to have an arbitrarily high,
very thin peak: at some observation. D
Given a partitioned parameter (e, 1]), it is sametimes helpful to consider the profile
like lihood. Given a likelihood lik n (e, 1]) (X 1, . . . , X n), the profile likelihood for e is defined
as the function
e sup lik n ( e, 1]) (X 1, . . . , X n) .
rJ
The supremum is taken over all possible values of 1]. The point of maximum of the profile
likelihood is exactly the first coordinate of the maximum likelihood estimator (e, f}). We
are simply computing the maximum of the likelihood over (e, 1]) in two steps.
It is rarely possible to compute a profile likelihood explicitly, but its numerical evaluation
is often feasible. Then the profile likelihood may serve to reduce the dimension of the
likelihood function. Profile likelihood functions are often used in the same way as (ordinary)
likelihood functions of parametric models. Apart from taking their points of maximum as
estimators e, the second derivative at e is used as an estimate of minus the inverse of the
asymptotic covariance matrix of e. Recent research appears to validate this practice.
25.69 Example (Cox model). Suppose that we observe arandom sample from the distri-
bution of X == (T, Z), where the conditional hazard function of the "survival time" T with
covariate Z takes the form
AT I zet) == e()Z A(t).
The hazard function A is completely unspecified. The density of the observation X == (T, Z)
is equal to
e()ZA(t)e-e()ZA(t) ,
where A is the primitive function of A (with A(O) == O). The usual estimator for (e, A)
based on a sample of size n from this model is the maximum likelihood estimator (e, A),
where the likelihood is defined as, with A {t} the jump of A at t,
n
(e, A) fl e()Zi A {ti }e-e()Zi A(td.
i=1
This is the product of the density at the observations, but with the hazard function A(t)
replaced by the jumps A {t} of the cumulative hazard function. (This likelihood is cIo se
404
Semiparametric Models
but not exactly equal to the empiricallikelihood of the model.) The form of the likelihood
forces the maximizer A to be a jump function with jumps at the observed "deaths" ti, only
and hence the likelihood can be reduced to a function of the unknowns A {tI}, . . . , A {t n }. It
appears to be impossible to derive the maximizers (e, A) in closed-form formulas, but we
can make some headway in characterizing the maximum likelihood estimators by "profiling
out" the nuisance parameter A. Elementary calculus shows that, for a fixed (), the function
n
(JI.], . .. , An) r+ n e8Z ; Ai e -eO'; Lj"j"Y; Aj
i=I
is maximal for
= L e 8z ;.
Ak .
l : ti .tk
The profile likelihood for () is the supremum of the likelihood over 1'_ for fixed (). In view
of the preceding display this is given by
n e z .
e I
e r+ n L 8z e - 1 .
'- I J .. t .> t .e)
l- . )_ I
The latter expression is known as the Cox partiallikelihood. The original motivation for
this criterion function is that the terms in the product are the conditional probabilities that
the ith subject dies at time i given that one of the subjects at risk dies at that time. The
maximum likelihood estimator for A is the step function with jumps
A 1
A { tk} == " .
""' , . ee Zi
l . ti ?::.tk
The estirnators e and A are asymptoticall y efficient, under some restrictions. (See sec-
tion 25.12.1.) We note that we have ignored the fact that jumps of hazard functions are
smaller than 1 and have maximized over all measures A. D
25.70 Example (Scale mixture). Suppose we observe a sample from the distribution of
X == () + Z8, where the unobservable variables Z and 8 are independent with completely
unknown distribution 17 and a known density ep, respectively. Thus, the observation has a
mixture density f pe (x Iz) d17(z) for the kernel
1 ( X - () )
pe(x Iz) == ep z .
If ep is syrnmetric about zero, then the mixture density is symmetric about (), and we
can estimate () asymptotically efficiently with a fully adaptive estimator, as discussed in
Section 25.8.1. Alternatively, we can take the mixture form of the underlying distribution
into account and use, for instance, the maximum likelihood estimator, which maxirnizes
the likelihood
(e, 17) r+ f1 J P8 (Xi Iz) d17(z).
i=I
Under some conditions this estimator is asyrnptotically efficient.
25.10 Maximum Likelihood Estimators
405
Because the efficient score function for f3 equals the ordinary score function for f3,
the maximum likelihood estimator satisfies the efficient score equation ]Pn-ee,7J == O. By
the convexity of the model in 17, this equation is unbiased in 17. Thus, the asymptotic
efficiency of the maximum likelihood estimator e follows under the regularity conditions
of Theorem 25.54. Consistency of the sequence of maximum likelihood estimators (en, f]n)
for the product of the Euclidean and the weak topology can be proved by the method of
Wald. The verification that the functions le,7J form a Donsker class is nontrivial but is
possible using the techniques of Chapter 19. O
25.71 Example (Penalized logistic regression). In this model we observe arandom sam-
ple from the distribution of X == (V, W, Y), for a 0-1 variable Y that follows the logistic
regres sion model
P Đ, 7J (Y == 1 IV, W) == \II ( f3 V + 17 (W) ) ,
where \II(u) == 1j(1 + e- U ) is the logistic distribution function. Thus, the usuallinear
regres sion of (V, W) has been replaced by the partiallinear regression f3 V + 17(W), in
which 1] ranges over a large set of "smooth functions." For instance, 17 is restricted to the
Sobolev class of functions on [O, 1] whose (k - l)st derivative exists and is absolutely
continuous with J (17) < 00, where
J2(T) = 11 (T)(k) (w»)2 dw.
Here k > 1 is a fixed integer and 17(k) is the kth derivative of 17 with respect to z.
The density of an observation is given by
pe,7J(X) == W(f3v + 1](w))Y(l - \II(f3v + 1](w))l- Y fv,w(v, w).
We cannot use this directly for defining a likelihood. The resulting maximizer f] would be
such that f](Wi) == 00 for every Wi with Yi == 1 and f](Wi) == -00 when Yi == O, or at least
we could construct a sequence of finite, smooth 1]m approaching this extreme choice. The
problem is that qualitative smoothness assumptions such as J (17) < 00 do not restrict 17 on
a finite set of points Wl, . . . , W n in any way.
To remedy this situation we can restrict the maximization to a smaller set of 17, which we
allow to growas n 00; for instance, the set of all 1] such that J (17) < M n for M n t 00
at a slow rate, or a sequence of spline approximations.
An alternative is to use a penalized likelihood, of the form
,,2 2
(f3, 1]) ]p n log Pe, 7J - An J (17).
Here n is a "smoothing parameter" that determines the importance of the penalty J2 (17). A
large value of n leads to smooth maximizers f], for small values the maximizer is more like
the unrestricted maximum likelihood estimator. Intermediate values are best and are often
chosen by a data-dependent scheme, such as cross-validation. The penalized estimator e
can be shown to be asymptotically efficient if the smoothing parameter is constructed to
satisfY n == op(n- 1 / 2 ) and l == Op(n k /(2k+l») (see [102]). O
25.72 Example (Proportionalodds). Suppase that we observe arandom sample from
the distribution of the variable X == (T 1\ C, 1 {T < C}, Z), in which, given Z, the variables
406
Semiparametric Models
T and C are independent, as in the random censoring model, but with the distribution
function F (t I z) of T given Z restricted by
F(t Iz) zTe
1 - F(t Iz) = e TJ(t).
In other words, the conditional odds given z of survival until t follows a Cox-type regres sion
model. The unknown parameter 17 is a nondecreasing, cadlag function from [O, (0) into
its elf with 17 (O) == O. It is the odds of survival if () == O and T is independent of Z.
If 17 is absolutely continuous, then the density of X == (Y, Ll, Z) is
( e- ZTe17 '(Y) (1 - Fc(Y - I Z)) ) 8 ( e-zTe fc(Y I z) ) 1-0
2 Te fz(z).
(17(Y) + e- zTe ) 17(Y) + e- Z
We cannot use this density as a likelihood, for the supremum is infinite unless we restrict rJ
in an important way. Instead, we view rJ as the distribution function of a measure and use
the empiricallikelihood. The probability that X == x is given by
( e-ZTe17{Y} (1 - Fc(Y - Iz)) ) 8 ( e-ZTe Fc({y} I Z) ) 1-0
(TJ(Y) + e-zT(j)(TJ(Y-) + e-zT(j) TJ(Y) + e-zT(j Fz{z},
For likelihood inference concerning ((), rJ) only, we may drop the terms involving Fc and
F z and define the likelihood for one observation as
( T ) o ( T ) I-o
. e- z e 17 {Y} e- Z e
lIk ( (), rJ) (x) == T T T '
(17(Y) + e- Z e)(17(Y-) + e- Z e) ry(y) + e- Z e
The presenee of the jumps rJ {y} causes the maximum likelihood estimator fJ to be a step
function with support points at the observed survival times (the values Yi corresponding to
8 i == 1). First, it is clear that each of these points must receive apositive mass. Second,
mass to the right of the largest Yi such that 8 i == 1 can be deleted, meanwhile increasing the
likelihood. Third, mass assigned to other points can be moved to the closest Yi to the right
such that 8 i == 1, again increasing the likelihood. If the biggest observation Yi has 8 i == 1,
then fJ {Yi} == 00 and that observation gives a contribution 1 to the likelihood, because the
function p 1---+ P / (p + r) attains for p > O its maximal value 1 at p == 00. On the other
hand, if 8 i == O for the largest Yi, then all jumps of fJ must be finite.
The maximum likelihood estimators have been shown to be asymptotically efficient
under some conditions in [105]. O
25.10.1 Random Censoring
Suppose that we observe arandom sample (XI, llI), ..., (X n , Ll n ) from the distribution
of (T /\ C, 1 {T < C}), in which the "survival time" T and the "censoring time" C are
independent with completely unknown distribution functions F and G, respectively. The
distribution of a typical observation (X, Ll) satisfies
PF,G(X < x,!l=O)= r (I-F)dG,
J[O,x]
PF,G(X < x, !l = 1) = r (1 - G_) dF.
J[O,x]
25.10 Maximum Likelihood Estimatars
407
Consequently, if F and G have densities f and g (relative to some dominating measures ),
then (X, ) has density
(x, 8) ((1 - F)(x)g(x))O ((1 - G-)(x)f(x) )1-0.
For f and g interpreted as Lebesgue densities, we cannot use this expression as a factor in
a likelihood, as the resulting criterion would have supremum infinity. (Simply choose f or
g to have a very high, thin peak at an observation Xi with i == 1 or i == O, respectively.)
Instead, we may take f and g as densities relative to counting measure. This leads to the
empiricallikelihood
n n
(F, G) 0((1- F)(Xi)G{Xi})I- iO((I- G_)(Xi)F{Xi}) i.
i=I i=I
In view of the product fonn, this factorizes in likelihoods for F and G separately. The
maximizer F of the likelihood F Il7=I(1 - F)(Xi)l- iF{Xi} i tums out to be the
product limit estimator, given in Example 20.15.
That the product limit estimator maximizes the likelihood can be seen by direct ar-
guments, but a slight detour is more insightfu1. The next lemma shows that under the
present model the distribution PF,G of (X, ) can be any distribution on the sample space
[O, (0) X {O, I}. In other words, if F and G range over all possible probability distributions on
[0,00], then pp,G ranges over all distributions on [O, (0) x {O, I}. Moreover, therelationship
(F, G) +* P p ,Gisone-to-oneontheintervalwhere(I-F)(I-G) > O. Asaconsequence,
there exists a pair (F, G) such that pp,G is the empirical distribution JP n of the observations
Pp,6{X i , i} == JPn{X i , i},
1 < i < n.
Because the empirical distribution maximizes P Il7=I P{X i , i} over all distributions,
it follows that (P, G) maximizes (F, G) Il 7= 1 PF,G{X i , i} over all (F, G). That F is
the product limit estimator next follows from Example 20.15.
Tocompletethediscussion, westudythemap (F, G) +* PF,G. Aprobabilitydistribution
on [O, (0) x {O, 1} can be identified with a pair (Ho, HI) of subdistribution functions on
[O, (0) such that Ho(oo) + HI (00) == 1, by letting Hi (x) be the mass of the set [O, x] x {i}. A
given pair of distribution functions (Fo, F 1 ) on [O, (0) yields such a pair of subdistribution
functions (Ho, HI), by
Ho(x) == [ (1 - F 1 ) dFo,
J[O,x]
HI (x) == [ (1 - F o -) dF I .
J[O,x]
(25.73)
Conversely, the pair (Fo, F I ) can be recovered from a given pair (Ho, HI) by, with Hi the
jump in Hi, H == Ho + HI and Af the continuous part of Ai,
i dHo 1 dHI
Ao(x) == , AI(X) == ,
[O,x] 1 - H_ - HI [O,x] 1 - H_
I - F i (x) == O (1 - Ai { S } ) e - Af (x) .
o::::s::::x
25.74 Lemma. Given any pair (Ho, HI) ofsubdistributionfunctions on [O, (0) such that
Ho(oo) + HI (00) == 1, the preceding display defines a pair (Fo, F I ) of subdistribution
functions on [O, (0) such that (25.73) holds.
408
Semiparametric Models
Proof. For any distribution function A and cumulative hazard function B on [O, (0), with
B C the continuous part of B,
1 - A(t) == Il (1 - B{s})e-BC(t) iff B(t) == 1 dA .
O s t [O,t] 1 - A_
To see this, rewrite the second equality as (1 - A_) dB == dA and B(O)
integrate this to rewrite it again as the Volterra equation
A(O), and
(1- A) == 1 + 1 (1- A_)d(-B).
[O,, ]
It is well known that the Vo1terra equation has the first equation of the display as its unique
solution. t
Combined with the definition of F ž , the equivalence in the preceding display implies
immediately that dA ž == dF ž /(l - F ž -). Secondly, as immediate consequences of the
definitions,
(1 - Fo)(l - Fl)(t) == Il (1 - Ao - Al + Ao Al)(s)e-(Ao+Al)C(t),
s t
1 dH
(Ao + Al)(t) - L Ao(s) Al(S) == .
s t [O,t] 1 - H_
(SplitdHo/ (1- H_ - Hl) into the parts corresponding to dH and Ho and note that Hl
may be dropped in the first p art. ) Combining these equations with the Volterra equation,
we obtain that 1 - H == (1 - Fo)(l - F 1 ). Taken together with dH 1 == (1 - H_) dA 1 ,
we conclude that dH 1 == (1 - F o -)(l - F 1 -) dA 1 == (1 - FO-) dF 1 , and similarly dHo ==
(1 - F 1 ) dFo. .
25.11 Approximately Least-Favorable Submodels
If the maximum likelihood estimator satisfies the efficient score equation JIDnle,ry == O,
then Theorem 25.54 yields its asymptotic normality, provided that its conditions can be
verified for the maximum like]ihood estimator f}. Somewhat unexpectedly, the efficient
score function may not be a "proper" score function and the maximum likelihood estimator
may not satisfy the efficient score equation. This is because, by definition, the efficient score
function is a projection, and nothing guarantees that this projection is the derivative of the
log likelihood along some submodel. If there exists a "Ieast favorable" path t 1--+ 17t (e, f})
such that 170 (e, f}) == f}, and, for every x,
- a ( )
f e , ry (x) == - log lik e + t, 17 t (e, f}) (x),
at It=O
then the maximum likelihood estimator satisfies the efficient score equation; if not, then this
is not clear. The existence of an exact least favorable submodel appears to be particularly
uncertain at the maximum likelihood estimator (e, f}), as this tends to be on the "boundary"
of the parameter set.
t See, for example, [133, p. 206] or [55] for an extended diseussion.
25.11 Approximately Least-Favorable Submodels
409
A method around this difficulty is to replace the efficient score equation by an approxi-
mation. First, it suffices that (e, ) satisfies the efficient score equation approximately, for
Theorem 25.54 goes through provided,Jii Pnle,f} == o p (1). Second, it was noted following
the proof of Theorem 25.54 that this theorem is valid for estimating equations of the form
PWle,1] == O for arbitrary mean-zero functions le,r]; its assertion remains correct provided
that at the true value of (f), 77) the function le, r] is the efficient score function. This suggests
to replace, in our proof, the function le,r] by functions Ke,r] that are proper score functions
and are close to the efficient score function, at least for the true value of the parameter.
These are derived from "approximately-least favorable submodels."
We define such submodels as maps t 1--+ rJt (e, rJ) from a neighborhood of O E R k to the
parameter set for 17 with rJa (e, rJ) == rJ (for every (f), rJ)) such that
a
K e , r] (x) == - log lik (f) + t, rJ t ( f), r])) (x ) ,
at It=O
exists (for every x) and is equal to the efficient score function at (f), 17) == (e o , rJo). Thus,
the path t 1--+ 17t (e, 17) must pass through rJ at t == O, and at the true parameter (e o , rJo) the
submodel is truly least favorable in that its score is the efficient score for e. We need such
a submodel for every fixed (f), 17), ar at least for the true value (e o , rJo) and every possible
value of (e, 17).
If (e, 17) maximizes the likelihood, then the function t 1--+ IP n log lik( e + t, 17t (e, ))
is maximal at t == O and hence (e, 17) satisfies the stationary equation PnKe,1] == O. Now
Theorem 25.54, with le,r] replaced by Ke,r]' yields the asymptotic efficiency of en. For easy
reference we reformulate the theorem.
Pen,r]OKen,f}n == o p (n- 1 / 2 + lien - e a II)
P eO ,1]O II K en ,f}1l - Keo,r]o 11 2 O, P en ,1]O II Ken,f}n 11 2 == Op(l).
(25.75)
(25.76)
25.77 Theorem. Suppose that the model {P e ,1] : e E el, is differentiable in quadratic
mean with respect to () at (e a , rJo) and let the efficient information matrix leo, r]O be nonsingu-
lar. Assume that Ke,r] are the scorefunctions ofapproximately least-favorable submodels (at
(f)o, 770)), that thefunctions Ke,f} belong to a Peo,r]o-Donskerclass with square-integrable en-
velope with probability tending to 1, and that (25.75) and (25.76) hold. Then the maximum
likelihood estimator en is asymptotically efficient at (e o , 170) provided that it is consistent.
The no-bias condition (25.75) can be analyzed as in (25.60), with le, 1] replaced by Ke,f}.
Alternatively, it may be useful to avoid evaluating the efficient score function at e ar , and
(25.60) may be adapted to
Pe n Ke n == (Pe n - Pe n)(K e n - Ke o no)
",o '" , "o ,', ,', ,.,
- / K8 0 , ryo [Pr!,ry - Pr!, ryo - B8 0 ' ryo (f} - 1)0) P8 0 , ryo] d J.L,
(25.78)
Replacing e by e o should make at most a difference of o p (\I e - e o II), which is negligible
in the preceding display, but the presence of 17 may require a rate of convergence for 17.
Theorem 5.55 yields such rates in same generality and can be translated to the present
setting as follows.
410
Semiparametric Models
Consider estimators in contained in a set Hn that, for a given n contained in a set
An C IR, maximize a criterion T r--+ JPnmTJ./l' or at least satisfy JPnmTJ.n > JPnmTo,l/l'
Assume that for every A E An, every T E Hn and every 8 > O,
P(mT,A - mTO,A);S - di(T, TO) + A 2 ,
E* sup I((;n (mT,A - mTO,A) I ;S CPn (8).
d)..(T,TO)<O
AEA/l,TEH n
(25.79)
(25.80)
25.81 Theorem. Suppase that (25.79) and (25.80) are valid for functions CPn such that
8 r--+ CPn (8) /8 a is decreasing for same ex < 2 and sets An x Hn such that P( n E An, in E
Hn) -+ 1. Then dl (in, TO) < O (8n + n) for any sequence of positive numbers 8n such
that CPn (8n) < ,Jfi 8 for every n.
25.11.1 Cox Regression with Current Status Data
Suppose that we observe arandom sample from the distribution of X == (C, , Z), in which
== 1 {T < C}, that the "survival time" T and the observation time C are independent
given Z, and that T follows a Cox model. The density of X relative to the product of Fc,z
and counting measure on {O, 1} is given by
( T ) o ( T ) I-o
Pe,A(X) == 1- exp(-ee ZA(c)) exp(-ee ZA(c)) .
We define this as the likelihood for one observation x. In maximizing the likelihood we
restrict the parameter e to a compact in IR k and restrict the parameter A to the set of all
cumulative hazard functions with A(i) < M for a fixed large constant M and T the end of
the study.
We make the following assumptions. The observation times C possess a Lebesgue
density that is continuous and positive on an interval [a, i] and vanishes outside this
interval. The true parameter Aa is continuously differentiable on this interval, satisfies
O < Aa (a -) < Aa ( T) < M, and is continuously differentiable on [a, TJ. The covariate
vec tor Z is bounded and E cov(Z I C) > O. The function heo,Ao given by (25.82) has a
version that is differentiable with a bounded derivative on [a, T]. The true parameter eo is
an inner point of the parameter set for e.
The score function for e takes the form
£e,A(X) == zA(c)Qe,A(x),
for the function Qe,A given by
[ _eeT Z A(c) ]
Qe.A (x) == ee T z 8 e T - (1 - 8) .
1 - e-ee .c A(c)
For every nondecreasing, nonnegative function h and positive number t, the sub model
At == A + this well defined. Inserting this in the log likelihood and differentiating with
respect to t at t == O, we obtain a score function for A of the form
Be.Ah(x) == h(C)Qe,A(X).
25.11 Approximately Least-Favorable Submodels
411
The linear span of these score functions contains B e , A h for all bounded functions h of
bounded variation. In view of the similar structure of the scores for f) and A, projecting
fe,A anto the closed linear span of the nuisance scores is a weighted least-squares problem
with weight function Qe,A. The solution is given by the vector-valued function
h ()-A( ) Ee,A(ZQ ,A(X)IC=c)
e,A c - c Ee,A(Q ,A(X) le = c) .
The efficient score function for f) takes the form
(25.82)
le,A(X) == (zA(c) - he,A(C))Qe,A(X).
Formally, this function is the derivative at t == O of the log likelihood evaluated at (f) + t, A-
t T he, A)' However, the second coordinate of the latter path may not define a nondecreasing,
nonnegative function for every t in a neighborhood of O and hence cannot be used to obtain
a stationary equation for the maximum likelihood estimator. This is true in particular for
discrete cumulative hazard functions A, for which A + th is nondecreasing for both t < O
and t > O only if h is constant between the jumps of A.
This suggests that the maximum likelihood estimator does not satisfy the efficient score
equation. To prove the asymptotic normality of e, we replace this equation by an approxi-
mation, obtained from an approximately least favorable submodel.
For fixed (f), A), and a fixed bounded, Lipschitz function ep, define
At(f), A) == A - t T ep (A) (heo,Ao o Aa l ) (A).
Then At (e, A) is a cumulative hazard function for every t that is sufficiently close to zero,
because for every u < v,
At(e, A)(v) - At(e, A)(u) > (A(v) - A(u)) (1 - IIt 1I11c/J heo,Ao o AOlIILiP)'
Inserting (f) + t, At(f), A)) into the log likelihood, and differentiating with respect to t at
t == O, yields the score function
Ke,A (x) = (zA(c) - c/J(A(c)) (heo,Ao o Aa l )(A(c))) Qe,A (x).
If evaluated at (e o , Ao) this reduces to the efficient score function leo,A o (x) provided
ep (Ao) == 1, whence the submodel is approximately least favorable. To prove the asymptotic
efficiency of en it suffices to verify the conditions of Theorem 25.77.
The function ep is a technical device that has been introduced in order to ensure that
O < At (e, A) < M for all t that are sufficiently close to O. This is guaranteed if O <
y ep (y) < c (y /\ (M - y) ) for every O < y < M, for a sufficiently large constant c. Because
by assumption [ Ao (o' - ), Ao ( T)] C (O, M), there exists such a function ep that also fulfills
ep (Ao) == 1 on [O', r].
In order to verify the no-bias condition (25.52) we need a rate of convergence for An.
25.83 Lemma. Under the conditions !isted previously, en is consistent and II An - Ao II Po,2 ==
O p (n- 1/3 ).
Proof. Denote the index (f)o, Ao) by O, and define functions
me,A == log (Pe,A + Po)/2.
412
Semiparametric Models
The densities Pe, A are bounded above by 1, and under our assumptions the density PO is
bounded away from zero. It follows that the functions me,A (x) are uniformly bounded in
( 8, A) and x .
By the concavity of the logarithm and the definition of (e, A),
IfDnme,A > IfD n log Pe,A + IfD n log PO > IfD n log Po == IfDnma.
Therefore, Theorem 25.81 is applicable with i == (8, A) and without A. For technical
reasons it is preferable first to establish the consistency of (e, A) by aseparate argument.
We apply Wald' s pro of, Theorem 5.14. The parameter set for 8 is compact by assumption,
and the parameter set for A is compact relative to the weak topology. Wald' s theorem
shows that the distance between (e, A) and the set of maximizers of the Kullback - Leibler
divergence converges to zero. This set of maximizers contains (8 0 , Aa), but this parameter
is not fully identifiable under our assumptions: The parameter Aa is identifiable only on
" p "p
the interval ((5, i). It follows that 8 --+ 8 0 and A (t) --+ Aa (t) for every (5 < t < i. (The
convergence of A at the points (5 and i does not appear to be guaranteed.)
By the proof of Lemma 5.35 and Lemma 25.85 below, condition (25.79) is satisfied with
d( (8, A), (80, Aa)) equal to 118 - 8 0 II + IlA - Aa 112. By Lemma 25.84 below, the bracketing
entropy of the class of functions me,A is of the order (1/8). By Lemma 19.36 condition
(25.80) is satisfied for
<PnClj) = J8(1 + o0n )'
This leads to a convergence rate of n- lj3 for both Ile - 8 0 II and II A - Aa 112. .
To verify the no-bias condition (25.75), we use the decomposition (25.78). The inte-
grands in the two term s on the right can both be seen to be bounded, up to a constant, by
(A - Aa)2, with probability tending to one. Thus the bias Pe,r]oKe,f} is actually of the order
O p (n- 2 / 3 ).
The functions x 1---+ Ke,A (x) can be written in the form 1/1 (z, ee T z, A(c), 8) for a function
1/1 that is Lipschitz in its first three coordinates, for 8 E {O, I} fixed. (Note that A 1--+ A Qe,A
is Lipschitz, as A 1--+ heo,A o o Aa l (A) lA == (heo,Aol Aa) o Aa l (A).) The functions z 1--+ z,
Z 1---+ exp 8 T Z, C 1---+ A (c) and 8 1---+ 8 form Donsker classes if 8 and A range freely. Rence
the functions x 1--+ A(c) Qe,A (x) form a Donsker class, by Example 19.20. The efficiency
of en follows by Theorem 25.77.
25.84 Lemma. Under the conditions listed previously, the e exists a constant C such
that, for every 8 > O,
lOgN[{S, {me,A' (e, A)}, L 2 (Po)) < c( ).
Proof. First consider the class of functions me, A for a fixed 8. These functions depend on
A monotonely if considered separately for 8 == O and 8 == 1. Thus a bracket Al < A < A 2
for A leads, by substitution, readily to a bracket for me,A' Furthermore, because this
dependence is Lipschitz, there exists a constant D such that
f (me,AI - me,A2)2 dFc,z < D ir (Al (e) - A 2 (e))2 de.
25.11 Approximately Least-Favorable Submodels
413
Thus, brackets for A of L 2 -size E translate into brackets for me,A of L 2 (Pe,A)-size propor-
tional to e. By Example 19.11 we can cover the set of all A by exp C(lje) brackets of
Slze 8.
Next, we allow eto vary freely as well. Because e is finite-dimensional and aj ae me,A (x)
is uniformly bounded in (e, A, x), this increases the entropy only slightly. .
25.85 Lemma. Under the eonditions listed previously there exist eonstants C, E > O such
that, for all A and all Ile - e o II < E,
f (p : - p :, y dJL > c i' (A - Ao)2(e) de + qe - e o 11 2 .
ProoJ. The left side of the lemma can be rewritten as
f (Pe,A - P e o,A o )2 d
1/2 1/2 2 J.L.
(Pe,A + Peo,Ao)
Because PO is bounded away from zero, and the densities Pe,A are uniformly bounded, the
denominator can be bounded above and below by positive constants. Thus the Hellinger
distance (in the display) is equivalent to the L 2 -distance between the densities, which can
be rewritten
! [ eT eT z ] 2
2 e-e ZA(c) _ e-e o Ao(c) dFy,z(e, z).
Let g(t) be the function exp( _ee T Z A(e)) evaluated at et == te + (1 - t)e o and At
tA + (1 - t)Ao, for fixed (e, z). Then the integrand is equal to (g(l) - g(O))2, and hence,
by the mean value theorem, there exists O < t == t (e, z) < 1 such that the preceding display
is equal to
Po (e-At(c)e 9 Tzee,rz [(A - Ao)(e)(1 + tee - eol z) + (e - eol zAo(e) J) 2,
T
Here the multiplicative factor e-At(c)ee t zeer Z is bounded away from zero. By dropping this
term we obtain, up to a constant, a lower bound for the left side of the lernma. N ext, because
the function Qeo,Ao is bounded away from zero and infinity, we may add a factor Q o,Ao'
and obtain the lower bound, up to a constant,
Po ((1 + tee - eol z)Beo,Ao (A - Ao)(x) + (e - eol leo,Ao (x) r.
Here the function h == (1 + t (e - eo)T z) is uniformly close to 1 if e is close to e o .
Furthermore, for any function g and vector a,
( T . ) 2 ( T . - ) 2
Po(Beo,Aog)a £eo,Ao == Po(Beo,Aog)a (£eo,A o - £0)
2 T -
< Po(Beo,Aog) a (lo - Io)a,
by the Cauchy-Schwarz inequality. Because the efficient information lo is positive-definite,
the term aT (lo - lo)a on the right can be written aT Ioae for a constant O < e < 1. The
lemma now follows by application of Lemma 25.86 ahead. .
414
Semiparametric Models
25.86 Lemma. Let h, gl and g2 be measurable functions such that Cl < h < C2 and
(Pglg2)2 < cPgf Pg for a constant c < 1 and constants Cl < 1 < C2 close to 1. Then
P(hg l + g2)2 > c(pgi + Pg ),
for a constant C depending on c, Cl and C2 that approaches 1 - -JC as Cl t 1 and C2 t 1.
Proof. We may first use the inequalities
(hg l + g2)2 > Cl hg; + 2hg l g2 + C:;l hg
== h(gl + g2)2 + (Cl - l)hg; + (1 - c:;l )hg
> cl(gi +2g l g 2 +g ) + (Cl -1)C2gi + (c:;l -1)g .
N ext, we integrate this with respect to P, and use the inequality for P gl g2 on the second
term to see that the left side of the lemma is bounded below by
Cl (pgi - 2 J cPgiPg + PgD + (Cl - 1)c2 P gi + (c 2 l - 1)c2Pg .
Finally, we apply the inequality 2xy < x 2 + y2 on the second term. .
25.11.2 Exponential Frailty
Suppose that the observations are arandom sample from the density of X == (U, V) given by
PO,ry(u, v) = f ze- zu (}ze- OZV dT/(z).
This is a density with respect to Lebesgue measure on the positive quadrant of JR2, and
we may take the likelihood equal to just the joint density of the observations. Let (en, n)
maXlmlze
n
(e, 'tJ) 1--* n P8, 77 (U i , Vi).
i=l
This estimator can be shown to be consistent, under some conditions, for the Euclidean and
weak topology, respectively, by, for instance, the method of Wald, Theorem 5.14.
The "statistic" 1/18 (U, V) == U + e V is, for fixed and known e, sufficient for the nuisance
parameter. Because the likelihood depends on 'tJ only through this statistic, the tangent
set 77 PPe,T} for 'tJ consists of functions of U + e V only. Furthermore, because U + e V is
distributed according to a mixture over an exponential family (a gamma-distribution with
shape parameter 2), the closed linear span of 77 PPe,T} consists of all mean-zero, square-
integrable functions of U + e V, by Example 25.35. Thus, the projection onto the closed
linear span of 77 P Pe,T} is the conditional expectation with respect to U + e V, and the efficient
score function for e is the "conditional score," given by
18,77 (x) == £8,77 (x) - E8 (£8,77 (X) 11/18 (X) == 1/18 (x))
f (u - e v )Z3 e- z (u+8v) d'tJ (z)
f e z 2 e - Z (u +8 v) d'tJ (z)
25.11 Approximately Least-Favorable Submodels
415
where we may use that, given U + 8 V == s, the variables U and 8 V are uniformly distributed
on the interval [O, s]. This function turns out to be also an actual score function, in that
there exists an exact least favorable subrnodel, given by
T]t(e, T])(B) = T](B(I- ;e )).
Inserting rJt (8, rJ) in the log likelihood, making the change of variables z (1 ..- t / (28)) --+ z,
and computing the (ordinary) derivative with respect to t at t == O, we obtain le, 1] (x). It
follows that the maximum likelihood estimator satisfies the efficient score equation, and its
asymptotic normality can be proved with the help of Theorem 25.54.
The linearity of the model in rJ (or the formula involving the conditional expectation)
implies that
P e , 1} O le,1] == O,
every 8,17,170.
Thus, the "no-bias" condition (25.52) is trivially satisfied. The verification that the functions
le,1} form a Donsker class is more involved but is achieved in the following lemma. t
25.87 Lemma. Suppose that J (Z2 + Z-5) d17o(Z) < 00. Then there exists a neighborhood
V of rJo for the weak topology such that the class offunctions
J (al + a2Z x + a3ZY) Z2 e- Z (b 1 x+b 2 y) d17(Z)
(x,y) J--+ J 2 (b b '
Z e- z lX+ 2Y) d17(Z)
where (al, . . . , a3) ranges over a bounded subset of JR?, (bI, b 2 ) ranges over a compact
subset of (O, (0)2, and rJ ranges over V, is P eO ,1}O -Donsker with square-integrable envelope.
25.11.3 Partially Linear Regression
Suppose that we observe arandom sample from the distribution of X
which for some unobservable error e independent of (V, W),
(V, W, Y), in
Y==8V+rJ(W)+e.
Thus, the independent variable Y is a regression on (V, W) that is linear in V with slope
8 but may depend on W in a nonlinear way. We assume that V and W take their values in
the unit interval [0,1], and that 17 is twice differentiable with J(17) < 00, for
J2(T]) = 11 T]1/(w)2dw.
This smoothness assumption should help to ensure existence of efficient estimators of 8
and will be us ed to define an estimatar.
If the (unobservable) error is assumed to be normal, then the density of the observation
X == (V, W, Y) is given by
P e (x ) == 1 e- 1 (y-eV-1}(w))2/ a 2 p v W ( v W ) .
,1] a,J2ii , ,
t For a proof see [106J.
416
Selniparametric Models
We cannot use this directly to define a maximum likelihood estimator for (e, rJ), as a
maximizer for rJ will interpolate the data exactly: A choice of 17 such that rJ (w ž ) == Yž - e V ž
for every i maximizes Il Pe, 17 (x ž ) but does not provide a useful estimator. The problem is
that so far rJ has only been restricted to be differentiable, and this does not prevent it from
being very wiggly. To remedy this we use a penalized log likelihood estimator, defined as
the minimizer of
(e, rJ) p JPn(y - ev - rJ(w»)2 + J2(rJ).
Here n is a "smoothing parameter" that may depend on the data, and determines the weight
of the "penalty" J2 (rJ). A large value of n gives much influence to the penalty term and
hence leads to a smooth estimate of rJ, and conversely. Intermediate values are best. For
the purpose of estimating e we may use any values in the range
2 == o ( n- 1 / 2 )
n p ,
1 == Op(n 2 / 5 ).
There are simple numerical schemes to compute the maximizer (en, f]n), the function f]n
being a natural cubic spline with knots at the values w 1, . . . , W n . The sequence en can be
shown to be asymptotically efficient provided that the regression components involving V
and W are not confounded or degenerate. More precisely, we assume that the conditional
distribution of V given W is nondegenerate, that the distribution of W has at least two
support points, and that ho(w) == E(V I W == w) has a version with J(h o ) < 00. Then, we
have the following lemma on the behavior of (en, f}n).
Let 1\.llw denote the normof L 2 (P W ).
25.88 Lemma. Under the conditions listed previously, the sequence en is consistent for
eo, II f]n 1100 == O p (1), J (f]n) == O p (1), and II f]n - rJ II w == O P ( n)' under (e o , rJo).
Proof. Write g(v, w) == ev + rJ(w), let JP n and Po be the empirical and true distribution
of the variables (e ž, V ž , W ž ), and define functions
mg,A(e, v, w) == (y - g(v, w»)2 + A2(J2(rJ) - J2(rJO))'
Then g(v, w) == ev + f](w) minimizes g p JPnmg,l' and
mg,A - mgO,A == 2e(go - g) + (go - g)2 + A 2 J2(r]) - A 2 J 2 (rJo).
By the orthogonality property of a conditional expectation and the Cauchy-Schwarz in-
equality, (EVrJ(W»2 < EE(V I W)2ErJ2(W) < EV21IrJlI . Therefore, by Lemma 25.86,
Po(g - gO)2 2: le - e o 1 2 + II rJ - rJo II .
Consequently, because poe == O and e is independent of (V, W),
PO(mg,A - mgO,A) 2: le - e o l 2 + 1117 - rJoll + A 2 J2(rJ) - A 2 .
This suggests to apply Theorem 25.81 with T == (e, rJ) and d ('r, TO) equal to the sum of
the first three term s on the right.
Because 1 == O p (1/ An) for An == n - 2/5, it is not a realloss of generality to assume
that n E An == [An, (0). ThendA(T,To) < 8 andA E An impliesthat le -eol < 8, that
II rJ - rJo II w < 8 and that J (rJ) < 8/ An. Assume first that it is known already that I e I and
25.11 Approximately Least-Favorable Submodels
417
II f] 1100 are bounded in probability, so that it is not a realloss of generality to assume that
le I v II f] 1100 < 1. Then
" le1]l m
po( e 1e1J1 - 1 - le1]l) == Po , < P 0 1] 2 Ee lel .
2 m.
m
Thus a bound on the II . II w-norm of 1] yields a bound on the "Bemstein norm" of e1] (given
on the left) of proportional magnitude. A bracket [1]1, 1]2] for 1] induces a bracket [e+1]1 -
e-1]2, e+TJ2 - e-1]I] for the functions e1]. In view of Lemma 19.37 and Example 19.10, we
obtain
( Jn (8) )
E sup IG n e(1] - 1]0) I ;S q;n (8) := In (8) 1 + 2.jii ,
d A (T,TO)<8 8 n
for
l n (8) = 1 8
( ) 1/2
1 + 8/An < 3/4 8
de rv 8 +1/4'
e An
This bound remains valid if we replace 1] -1]0 by g - go, for the parametric part e v adds little
to the entropy. We can obtain a similar maximal inequality for the process G n (g - gO)2,
in view of the inequality Po(g - gO)4 < 4P o (g - gO)2, still under our assumption that
le Iv 111]1100 < 1. We conclude that Theorem 25.81 applies and yields the rate of convergence
le - 8 0 1 + 11f] - 1]ollw == Op(n- 2 / 5 + ln) == Op(ln).
Finally, we must prove that e and II f] 1100 are bounded in probability. By the Cauchy-
Schwarz inequality, for every w and 1],
11](W) -1](0) -1]'(O)wl < 1 w 1 u 11]"I(s)dsdu < 1(1]).
This implies that 111] 1100 < 11] (O) I + 11]' (O) I + J (1]), whence it sufficies to show that e, f] (O),
f]' (O), and J (f]) remain bounded. The preceding display implies that
lev + 1](0) + 1]/(O)wl < Ig(v, w)1 + J(1]).
The empirical measure applied to the square of the left side is equal to aT An a for a ==
(e, TJ (O), 1]' (O)) and An == :rp> n (v, 1, w) (v, 1, w) T the sample second moment matrix of the
variables (Vi, 1, W i ). By the conditions on the distribution of (V, W), the corresponding
population matrix is positive-definite, whence we can conclude that a is bounded in prob-
ability as soon as aT Ana is bounded in probability, which is certainly the case if IfD n g 2 and
J (f]) are bounded in probability.
We can prove the latter by applying the preceding argument conditionally, given the
sequence VI, W 1 , V 2 , W 2 , . . . . Given these variables, the variables ei are the only random
part in mg).. - mga,).. and the parts (g - gO)2 only contribute to the centering function. We
apply Theorem 25.81 with square distance equal to
d Cr:, TO) == IfDn(g - go)2 + A 2 J2(1]).
An appropriate maximal inequality can be derived from, for example Corollary 2.2.8 in
[146], because the stochastic process Gneg is sub-Gaussian relative to the L 2 ( IfDn)-metric
on the set of g. Because d)..(T, TO) < 8 implies that IfDn(g - gO)2 < 8 2 , J(1]) < 8/An, and
le 1 2 v 1\ TJ II < c ( ]p n (g - gO) 2 + J2 (1])) for C dependent on the smallest eigenvalue of
418
Semiparametric Models
the second moment matrix An' the maximal inequality has a similar form as before, and we
conclude that JtDn(g - gO)2 + 2 J2(f]) == Op( 2). This implies the desired result. .
The normality of the error e motivates the least squares criterion and is essential for
the efficiency of e. However, the penalized least -squares method makes sense also for
nonnormal error distributions. The preceding lemma remains true under the more general
condition of exponentially small error tails: Ee c1el < 00 for same c > O.
Under the normality assumption (with eJ == 1 for simplicity) the score function for f) is
given by
fe,r](x) == (y - f)v - TJ(w))v.
Given a function h with J (h) < 00, the path rJt == TJ + t h defines a submodel indexed by
the nuisance parameter. This leads to the nuisance score function
Be,r]h(x) == (y - f)v - TJ(w))h(w).
On comparing these expressions, we see that finding the projection of le,r] onto the set of
TJ-scores is a weighted least squares problem. By the independence of e and (V, W), it
follows easily that the projection is equal to Be,r]ho for ho(w) == E(V I W == w), whence
the efficient score function for f) is given by
le,r](x) == (y - f)v - TJ(w)) (v - ho(w)).
Therefore, an exact least-favorable path is given by TJt(f), TJ) == TJ - tho.
Because (en, f]n) maximizes a penalized likelihood rather than an ordinary likelihood, it
certainly does not satisfy the efficient score equation as considered in section 25.8. However,
it satisfies this equation up to a term involving the penalty. Inserting (e + t, TJt({), f])) into
the least-squares criterion, and differentiating at t == O, we obtain the stationary equation
- /\2 (/! /!
J!DnlO,ij - 2A. lo 1 r, (w)ho(w) dw = 0,
The second term is the derivative of 2 J 2 (TJt(e, f])) at t == O. By the Cauchy-Schwarz
inequality, it is bounded in absolute value by 2 2 J(f])J(h o ) == op(n- 1 / 2 ), by the first as-
sumption on and because J(f]) == Op(l) by Lemma 25.88. We conclude that (en, f]n)
satisfies the efficient score equation up to a o p (n -1/2) -term. Within the context of Theo-
rem 25.54 a remainder term of this small order is negligible, and we may use the theorem
to obtain the asymptotic normality of {)n.
A formulation that also allows other estimators f] is as follows.
25.89 Theorem. Let f]n be any estimators such that II f]n 1100 == O p (1) and J (f}n) == O p (1).
Then any consistent sequence of estimators ()n such that -Jn JtD n le, fJ == o p (1) is asymptoti-
cally efficient at (f)o, TJo).
Proof. It suffices to check the conditions of Theorem 25.54. Since
Pe,r]ie,fJ == Pe,r] (TJ(w) - f}(w)) (v - ho(w)) == O,
for every (f), TJ), the no-bias condition (25.52) is satisfied.
25.12 Likelihood Equations
419
That the functions le, are contained in a Donsker class, with probability tending to 1,
follows from Example 19.10 and Theorem 19.5.
The remaining regularity conditions of Theorem 25.54 can be seen to be satisfied by
standard arguments. .
In this example we use the smoothness of 1] to de fine a penalized likelihood estimator
for e. This automatically yields a rate of convergence of n -2/5 for fJ. However, efficient
estimators for e exist under weaker smoothness assumptions on 1], and the minimal smooth-
ness of TJ can be traded against smoothness of the function g ( w) == E (V I W == w), which
also appears in the formula for the efficient score function and is unknown in practice. The
trade-off is a consequence of the bi as Pe,rJ,gle,i],g being equal to the cross product of the
biases in fJ and g. The square terms in the second order expansion (25.60), in which the
derivative relative to (1], g) (instead of 1]) is a (2 x 2)-matrix, vanish. See [35] for a detailed
study of this model.
25.12 Likelihood Equations
The "method of the efficient score equation" isolates the parameter e of interest and charac-
terizes an estimator e as the solution of a system of estimating equations. In this system the
nuisance parameter has been replaced by an estimator fJ. If the estimator fJ is the maximum
likelihood estimator, then we may hope that a solution e of the efficient score equation is
also the maximum likelihood estimator for e, or that this is approximately true.
Another approach to proving the asymptotic normality of maximum likelihood estimators
is to design a system of likelihood equations for the parameter of interest and the nuisance
parameter jointly. For a semiparametric model, this necessarily is a system of infinitely
many equations.
Such a system can be analyzed much in the same way as a finite-dimensional system.
The system is linearized in the estimators by a Taylor expansion around the true parameter,
and the limit distribution involves the i verse of the derivative applied to the system of
equations. However, in most situations an ordinary pointwise Taylor expansion, the classical
argument as employed in the introduction of section 5.3, is impossible, and the argument
must involve some advanced tools, in particular empirical processes. A general scheme is
given in Theorem 19.26, which is repeated in a different notation here. A limitation of this
approach is that both e and i) must converge at ,Jn-rate. It is not clear that a model can
always appropriately parametrized such that this is the case; it is certainly not always the
case for the natural parametrization.
The system of estimating equations that we are looking for consists of stationary equa-
tions resulting from varying either the parameter e or the nuisance parameter 1]. Suppose
that our maximum likelihood estimator (e, fJ) maximizes the function
(e, 1]) I1lik(e, 1])(X i ),
for lik ( e, 1]) (x) being the "likelihood" given one observation x.
The parameter e can be varied in the usual way, and the resulting stationary equation
takes the fonn
JID wee " == O.
,rJ
420
Semiparametric Models
This is the usual maximum likelihood equation, except that we evaluate the score function
at the joint estimator (e, f]), rather than at the single value e. A precise condition for this
equation to be valid is that the partial derivative of log like e, 17) (x) with respect to e exists
and is equal to £e,1] (x), for every x, (at least for 17 == f] and at e == e).
Varying the nuisance parameter 17 is conceptually more difficult. Typically, we can use
a selection of the submodels t 1---+ 17t used for defining the tangent set and the information
in the model. If scores for 17 take the form of an "operator" B(), 1] working on a set of indices
h, then a typicallikelihood equation takes the form
PnBe Ah == Pe ABe Ah.
,1] ,1],1]
Here we have made it explicit in our notation that a score function always has mean zero,
by writing the score function as x 1---+ B e ,1]h(x) - Pe,1]Be,1]h rather than as x 1---+ Be,17h(X).
The preceding display is valid if, for every (e, 1]), there exists some path t 1---+ 1]t (e, 17) such
that 1]0 (e, 1]) == 1] and, for every x,
a
B e ,1]h(x) - Pe,1] B e,1]h == - loglik(e + t, 1]t(e, 1])).
at It=O
Assume that this is the case for every h in some index set H, and suppose that the latter is
chosen in such away that the map h 1---+ Be,17h(X) - Pe,17Be,17h is uniformly bounded on H,
for every x and every (e, 1]).
Then we can define random maps \lin : k x H 1---+ k x fOO(H) by \lin == (\lin!, \lI n2 )
with
\lin 1 (e, 1]) == P n f e ,1]'
\lI n2 (e, 1])h == Pn B e,1]h - Pe,1] B e,1]h,
h E 'li.
The expectation of these maps under the parameter (e o , 1]0) is the deterministic map \li ==
(\li 1, \lI2) given by
\llI (e, 1]) == P eO ,1] O f e ,1]'
\li 2 ( e, 1]) h == P e o , 1]0 B e , 1] h - P e , 1] B e , 17 h ,
h EH.
By construction, the maximum likelihood estimators (en, fin) and the "true" parameter
(e o , 1]0) are zeros of these maps,
\lin (en, f]n) == O == \lI(e o , 1]0).
The argument next proceeds by linearizing these equations. Assume that the parameter
set H for 1] can be identified with a subset of a Banach space. Then an adaptation of
Theorem 19.26 is as follows.
25.90 Theorem. Suppose that thefunctions f e ,1] and B e ,1]h, ifh ranges over 'li and (e, 1])
over a neighborhood of (e o , 1]0), are contained in a P eO ,1]O -Donsker class, and that
P eO ,1]O II £e,17 - £e O ,1]O 11 2 O,
sup P eO ,1]O I Be,17 h - Be O ,1] O h 1 2 O.
hE7-{
Furthermore, suppose that the map \li : 8 x H 1---+ k X foo (H) is Frechet-differentiable at
(e o , 1]0), with a derivative W O :}Rk x lin H f---+ k x fOO('li) that has a continuous inverse
25.12 Likelihood Equations
421
on its range. If the sequence (en, f]n) is consistent for (e o , 1]0) and satisfies \lin (en, f]n) ==
o p (n -1/2), then
wo,Jn(e n - eo, f]n - TJo) == -,Jn\lln(e O , 170) + op(l).
The theorem gives the joint asymptotic distribution of en and f]n. Because ,Jn\lln (eo, 170)
is the empirical process indexed by the Donsker class consisting of the functions le O ,170
and BeO,170h, this process is asymptotically normally distributed. Because normality is
retained under a continuous, linear map, such as q., O- 1 , the limit distribution of the sequence
,Jn(e n - eo, n - 1]0) is Gaussian as well.
The case of a partitioned parameter (e, TJ) is an interesting one and illustrates most
aspects of the application of the preceding theorem. Therefore, we continue to write the
formulas in the corresponding partitioned form. However, the preceding theorem, applies
more generally. In Example 25.5.1 we wrote the score operator for a semiparametric model
in the form
T'
A e ,17(a, b) == a le,17 + Be,17b.
Corresponding to this, the system of likelihood equations can be written in the form
JP n A e ,17(a, b) == Pe,17Ae,17(a, b),
every (a, b).
If the partitioned parameter (e, TJ) and the partitioned "directions" (a, b) are replaced by
a general parameter i and general direction c, then this formulation extends to general
models. The maps \II n and \11 then take the forms
'¥nCr)c == JIDnATc - PTATg,
\II(i)C == PToATc - PTATc.
The theorem requires that these can be considered maps from the parameter set into a
Banach space, for instance a space £OO(C).
To gain more insight, consider the case that TJ is a measure on a measurable space
(Z, C). Then the directions h can often be taken equal to bounded functions h : Z IR,
corresponding to the paths d 17t == (1 + t h) d TJ if TJ is a completely unknown measure, or
dTJt == (1 + t(h - 1]h)) d17 if the total mass of each TJ is fixed .to one. In the remainder of
the discussion, we assume the latter. Now the derivative map \11 0 typically takes the form
( '¥U
(e - e o , 17 - TJo) .
\11 21
12 ) ( e - e o )
\11 22 1] - 170
where
. . 'T
\11 11 (e - eo) == - Peo,17oleo,17oleO,170 (e - eo),
W12(1] - 1]0) = - f B;O,ry),eO,ryO d(1] - 1]0),
. 'T
'¥21 (e - eo)h == - Pe o , 170 (Be o , 170 h )le o , 170 (e - e o ),
Wn(7J - I]o)h = - f B;O,ryOBeo,ryOh d(1] - 1]0),
(25.91)
422
Semiparametric Models
For instance, to find the last identity in an informal manner, consider a path rJt in the direction
of g, so that d rJt - d rJO == t g d rJO + o (t). Then by the definition of a derivative
W2(e O , rJt) - W2(eO, rJo) W22(rJt - rJo) + o(t).
On the other hand, by the definition of \II, for every h,
W2(e O , rJt)h - w(e o , rJo)h == -(Pe O ,1]t - PeO,1]o)BeO,1]th
-t Pe O ,1]O (B eO ,1]og) (B eO ,1] O h) + o(t)
= - f (B;o, oBlio, oh) tg drJo + o(t).
On comparing the preceding pair of displays, we obtain the last line of (25.91), at least for
drJ - drJo == g drJo. These arguments are purely heuristic, and this form of the derivative
must be established for every example. For instance, within the context of Theorem 25.90,
we may need to apply Wo to rJ that are not absolutely continuous with respect to rJO. Then the
validity of (25.91) depends on the version that is used to de fine the adjoint operator B e * 1] .
o, o
By definition, an adjoint is an operator between L 2 -spaces and hence maps equivalence
classes into equivalence classes.
The four partial derivatives \IIij in (25.91) involve the four parts of the information
operator Z,1]Ae,1]' which was written in a partitioned form in Exampl 25.5.1. In particular,
the map \11 11 is exactly the Fisher information for e, and the operator \11 22 is defined in term s
of the information operator for the nuisance parameter. This is no coineidenee, because
the formulas can be considered a version of the general identity "expectation of the second
derivative is equal to minus the information." An abstract form of the preceding argument
applied to the map w('r)c == Pr:oAr:c - Pr:Ar:c leads to the identity, with Tt a path with
derivative TO at t == O and score function Ar:od,
o (TO)C == (A;o Ar:o c, d) r:o'
In the case of a partitioned parameter T == (e, rJ), the inner inner product on the right is
defined as ((a, b), (a, fJ))r:o == aT a + f bfJ drJo, and the four formulas in (25.91) follow by
Example 25.5.1 and some algebra. A difference with the finite-dimensional situation is that
. .
the derivatives TO may not be dense in the domain of wo, so that the formula determines \ilo
only part1y.
An important condition in Theorem 25.90 is the continuous invertibility of the derivative.
Because a linear map between Euclidean spaces is automatically continuous, in the finite-
dimensional set-up this condition reduces to the derivative being one-to-one. For infinite-
dimensional systems of estimating equations, the continuity is far from automatic and may
be the condition that is hardest to verify. Because it refers to the ,eooCH)-norm, we have
some control over it while setting up the system of estimating equations and choosing the
set of functions H. A bigger set H makes OI more readily continuous but makes the
differentiability of W and the Donsker condition more stringent.
In the partitioned case, the continuous invertibility of \¥o can be verified by ascertaining
the continuous invertibility of the two operators 11 and V == 22 - 21 li 1 12. In that
case we have
. -1 _ ( U1 + lil 12 V-1 21 U1
\Ilo - -V-I 21 1i1
_ 1iI.\II12 V- I )
V- 1
25.12 Likelihood Equations
423
The operator \.lJ 11 is the Fisher information matrix for (J if rJ is known. If this would not be
invertible, then there would be no hope of finding asymptotically normal estimators for (J.
The operator V has the form
V(I) - I)o)h = - f (B;o'ryOBeo,ryo + K)h d(1) - Tlo),
where the operator K is defined as
( ( 'T ) 1 .
Kh == - P eO ,170 Be O ,17o h )leo,170 Ie ,17oB;o,17oleo,17o'
The operator V : lin H 1---+ loo (H) is certainly continuously invertible if there exists a
positive number E such that
sup\V(rJ - rJo)hl > eSllrJ - rJoll.
h EH
In the case that rJ is identified with the map h 1---+ rJh in loo (H), the norm on the right is
given by SUPhEH I (rJ - rJo)h I. Then the display is certainly satisfied if, for some eS > O,
{ (B;O'ryO BeO'ryO + K)h : h E 1i} :) EH
This condition has a nice interpretation if H is equal to the unit ball of a Banach space Iffi of
functions. Then the preceding display is equivalent to the operator B;0,170 B eO ,170 + K : 185 1---+ 185
being continuously invertible. The first part of this operator is the information operator for
the nuisance parameter. Typically, this is continuously invertible if the nuisance parameter
is regularly estimable at a ,Jn-rate (relatively to the norm used) if e is known. The following
lemma guarantees that the same is then true for the operator B;0,170 B()0,170 + K if the efficient
information matrix for (J is nonsingular, that is, the parameters e and rJ are not locally
confounded.
25.92 Lemma. Let Iffi be a Banach space contained in loo (L). If leo:17o is nonsingu-
lar, B;O,170Beo,170: Iffi 1---+ 185 is onto and continuously invertible and B;0,170l()0,170 E 185, then
B;o, 170 B()o, 170 + K : 185 1---+ Iffi is onto and continuously invertible.
Proof. Abbreviate the index (e o , rJo) to O. The operator K is compact, because it has a
finite-dimensional range. Therefore, by Lemma 25.93 below, the operator B; Bo + K is
continuously invertible provided that it is one-to-one.
Suppose that (B; Bo + K)h == O for some h E 185. By assumption there exists a path
t 1---+ rJt with score function Boh == Boh - PoBoh at t == O. Then the submodel indexed by
l' T' -
t 1---+ «(Jo + tao, rJt), for ao == -1 0 - Po(Boh)lo, has score function a o lo + Boh at t == O, and
information
T -2 T' -2 T
a o Ioao + Po(Boh) + 2a o Polo(Boh) == Po(Boh) - a o Ioao.
Because the efficient information matrix is nonsingular, this information must be strictly
positive, unless ao == O. On the other hand,
2 T .
O == rJoh(B Bo + K)h == Po(Boh) + a o Po(Boh)lo.
424
Semiparametric Models
This expression is at least the right side of the preceding display and is positive if aa f O.
Thus aa == O, whence K h == O. Reinserting this in the equation (B; Ba + K)h == O, we find
that B; Bah == O and hence h == O. .
The proof of the preceding lemma is based on the Fredholm theory of linear operators.
An operator K : IBS 1---+ IBS is compact if it maps the unit ball into a totally bounded set. The
following lemma shows that for certain operators continuous invertibility is a consequence
of their being one-to-one, as is true for matrix operators on Euclidean space. t It is also
useful to prove the invertibility of the information operator itself.
25.93 Lemma. Let IBS be a Banach space, let the operator A : IBS 1---+ IBS be continuous, onto
and continuously invertible and let K : IBS 1---+ JBS be a compact operator. Then R(A + K) is
closed and has codimension equal to the dimension ofN(A + K). In particular, if A + K
is one-to-one, then A + K is onto and continuously invertible.
The asymptotic covariance matrix of the sequence ,Jn(e n - ea) can be computed from
the expression for \lio and the covariance function of the limiting process of the sequence
,Jn \lin (e a , 170). However, it is easier to use an asymptotic representation of ,Jn (en - e a ) as
a sum. For a continuously invertible information operator B;0,170 B eO ,170 this can be obtained
as follows.
In view of (25.91), the assertion of Theorem 25.90 can be rewritten as the system of
equations, with a subscript Odenoting (e o , 170),
-la (en - ea) - (f7n - 17a)B la == -( JP n - Pa)la + Op(l/ ),
.T A
-Pa(Bah)fo (en - ea) - (f7n -17a)B Bah == -( JP n - Pa)Boh + op(l/ ).
The op(11 ,Jn)-term in the second line is valid for every h E Ji (uniformly in h). Ifwe can
also choose h == (B; Ba)-l B;la, and subtract the first equation from the second, then we
arrive at
[80,170 (en - e a ) == (JPn - P a )le O ,170 + op(l).
Rere leO,170 is the efficient score function for e, as given by (25.33), and [80,170 is the ef-
ficient information matrix. The representation shows that the sequence ,Jn (en - ea) is
asymptotically linear in the efficient influence function for estimating e. Renee the maxi-
mum likelihood estimator e is asymptotically efficient.:j: The asymptotic efficiency of the
estimator f7h for 17h follows similarly.
We finish this section with a number of examples. For each example we describe the
general structure and main points of the verification of the conditions of Theorem 25.90,
but we refer to the original papers for some of the details.
25.12.1 Cox Model
Suppose that we observe arandom sample from the distribution of the variable X
(T /\ C, 1 {T < C}, z), where, given Z, the variables T and C are independent, as in the
t For a proof see, for example, [132, pp. 99-103].
t This conclusion also can be reaehed from general resuits on the asymptotie effieieney of the maximum likelihood
estimator. See [56] and [143].
25.12 Likelihood Equations
425
random censoring model, and T folIows the Cox model. Thus, the density of X == (Y, , Z)
is given by
(ee z A(y)e-e oz A(y) (1 - Fc I z(y - Iz)) Y (e-e OZ A(y) Je I z(y Iz)) 1-8 pz(z).
We de fine a likelihood for the parameters (f), A) by dropping the factors involving the
distribution of (C, Z), and replacing A (y) by the pointmass A {y },
lik(f), A) (x) == (eeZ A {y }e-e 8Z A(Y)) 8 (e-e 8Z A(y)) 1-8.
This likelihood is convenient in that the profile likelihood function for f) can be computed
explicitly, exactly as in Example 25.69. Next, given the maximizer e, which must be
calculated numericalIy, the maximum likelihood estimator A is given by an explicit formula.
Given the general results put into place so far, proving the consistency of (e, A) is
the hardest problem. The methods of section 5.2 do not apply directly, because of the
empirical factor A {y} in the likelihood. These methods can be adapted. Altematively, the
consistency can be proved using the explicit fOffi1 of the profile likelihood function. We
omit a discussion.
For simplicity we make a number of partly unnecessary assumptions. First, we assume
that the covariate Z is bounded, and that the true conditional distributions of T and C given
Z possess continuous Lebesgue densities. Second, we assume that there exists a finite
number i > O such that P(C > i) == P(C == i) > O and Peo,Ao(T > i) > O. The latter
condition is not unnatural: It is satisfied if the survival study is stopped at some time i at
which apositive fraction of individuals is stilI "at risk" (alive). Third, we assume that, for
any measurable function h, the probability that Z #- hey) is positive. The function A now
matters only on [O, i]; we shalI identify A with its restriction to this interval. Under these
conditions the rnaximum likelihood estimator (e, A) can be shown to be consistent for the
product of the Euclidean topology and the topology of uniform convergence on [O, i].
The score function for () takes the form
le,A(x) == 8z - zeeZA(y).
For any bounded, measurable function h : [O, i] r-+ JR, the path defined by dAt == (1 +
th) dA defines a submodel passing through A at t == O. Its score function at t == O takes
the form
Be,Ah(x) == 8h(y) - eez 1 h dA.
[O,y]
The function h r-+ Be,Ah(x) is bounded on every set of uniformly bounded functions h,
for any finite measure A, and is even uniformly bounded in x and in (f), A) ranging over a
neighborhood of (f)o, Aa).
It is not difficult to find a formula for the adjoint B;,A of Be,A : L 2 (A) r-+ L 2 (P e ,A), but
this is tedious and not insightful. The information operator B;,A Be,A : L 2 (A) r-+ L 2 (A) can
be calculated from the identity Pe,A (Be,Ag) (Be,Ah) == Ag(B;,A Be,Ah). For continuous A
it takes the surprisingly simple form
B;,ABe,Ah(y) == hey) Ee,A ly yeez.
426
Semiparametric Models
To see this, write the product Be,Ag Be,Ah as the sum of four terms
oh(y)g(y) - oh(y)e lJz l Y g dA - og(y)e lJz l Y h dA + e 2IJz l Y g dA l Y h dA.
Take the expectation under Pe,A and interchange the order of the integrals to represent
Be,A Be,Ah also as a sum of four terms. Partially integrate the fourth term to see that
this cancels the second and third terms. We are left with the first term. The function
. .
Be,A £e,A, can be obtained by a similar argument, starting from the identity Pe,A £e,ABe,Ah ==
A(Be,A £e,A)h. It is given by
* . ez
Be,A £e,A == Ee,A 1 Y :::: y Ze .
The calculation of the information operator in this way is instructive, but only to check
(25.91) for this example. As in other examples a direct derivation of the derivative of the
map \11 == (\11 1 , \112) given by \11 1 (e, A) == Po£e,A and \11 2 (e, A)h == PoBe,Ah requires less
work. In the present case this is almost trivial, for the map \11 is already linear in A. Writing
Go (y I Z) for the distribution function of Y given Z, this map can be written as
'¥l(e, A) = EZe lJoZ 1 Go(y IZ) dAo(Y) - EZe lJZ 1 A(y) dGo(Y IZ),
\I1 2 ce, A)h == Eeeoz 1 h(y)Go(y I Z) dAo(Y) - Ee ez 11 h dA dGo(yl Z).
[O,y]
If we take 7-i equal to the unit ball of the space BV[O, i] of bounded functions of bounded
variation, then the map \11 : IR x £OO(H) 1---+ IR x £OO(H) is linear and continuous in A, and
its p arti al derivatives with respect to e can be found by differentiation under the expectation
and are continuous in a neighborhood of (e o , Ao). Several applications of Fubini's theorem
show that the derivative takes the form (25.91).
We can consider B Bo as an operator of the space BV[O, i] into itself. Then it is
continuously invertible if the function y 1---+ Eeo,Ao 1y::::yeeOz is bounded away from zero on
[O, i]. This we have (indirectly) assumed. Thus, we can apply Lemma 25.92. The efficient
score function takes the form (25.33), which, with Mi (y) == Eeo,Ao 1 y :::: y Zi eeoz, reduces to
leo,AO(X) == 8 ( Z - Ml (y) ) - ee o z 1 ( z - Ml (t) ) dAo(t).
Mo [O,y] Mo
The efficient information for e can be computed from this as
( ) 2
- e z Ml -
IlJo,Ao = Ee o 1 Z - Mo (y) Go(y I Z) dAo(Y).
This is strictly positive by the assumption that Z is not equal to a function of Y.
The class 7-i is a universal Donsker class, and hence the first parts 8 h (y) of the functions
Be,Ah form a Donsker class. The functions of the form f[o,y] h dA with h ranging over 1t
and A ranging over a collection of measures of uniformly bounded variation are functions of
uniformly bounded variation and hence also belong to a Donsker class. Thus the functions
Be,Ah form a Donsker class by Example 19.20.
25.12 Likelihood Equations
427
25.12.2 Partially Missing Data
Suppose that the observations are arandom sample from a density of the form
(X, y, z) f-+ ! pe(x I s) d17(S) Pe(Y Iz) d17(Z) =: pe(x 117) Pe(Y Iz) d17(Z).
Here the parameter 1] is a completely unknown distribution, and the kemel pe (. ls) is a given
parametric model indexed by the parameters () and s, relative to some density /-L. Thus,
we obtain equal numbers of bad and good (direct) observations concerning 1]. TypicalIy,
by themselves the bad observations do not contribute positive information conceming the
cumulative distribution function 1], but along with the good observations they help to cut
the asymptotic variance of the maximum likelihood estimators.
25.94 Example. This model can arise if we are interested in the relationship between a
response Y and a covariate Z, but because of the cost of measurement we do not observe Z
for a fraction of the population. For instance, a fulI observation (Y, Z) == (D, W, Z) could
consist of
- a logistic regres sion D on exp Z with intercept and slope {Ja and f3I, respectively, and
- a linear regres sion W on Z with intercept and slope aa and al, respectively, and an
N (O, a 2 )-error.
Given Z the variables D and W are assumed independent, and Z has a completely unspec-
ified distribution 17 on an interval in IR. The kernel is equal to, with \11 denoting the logistic
distribution function and ep denoting the standard normal density,
d I-dl ( W -aa -alZ )
pe(d, w Iz) = \If(f3o + f31 eZ ) (1 - W(f3o + f31 eZ )) (5 ep (5 .
The precise form of this density does not play a major role in the folIowing.
In this situation the covariate Z is a gold standard, but, in view of the costs of measure-
ment, for a selection of observations only the "surrogate covariate" W is available. For
instance, Z corresponds to the LDL cholesterol and W to total cholesterol, and we are
interested in heart disease D == 1. For simplicity, each observation in our set-up consists of
one fuH observation (Y, Z) == (D, W, Z) and one reduced observation X == (D, W). D
25.95 Example. If the kemel pe (y I z) is equal to the normal density with mean z and
variance (), then the observations are arandom sample ZI, . . . , Zn from 1], arandom sample
XI, . . . , X n from 1] perturbed by an additive (unobserved) normal error, and a sample
YI, . . . , Y n of random variables that given ZI, . . . , Zn are normalIy distributed with means
Zi and variance (). In this case the interest is perhaps focused on estimating 1], rather than
(). D
The distribution of an observation (X, Y, Z) is given by two densities and a nonparametric
part. We choose as likelihood
lik ( (), 1]) (x, y, z) == P e (x I 1]) P e (y Iz) 1] { z } .
Thus, for the completely unknown distribution 1] of Z we use the empiricallikelihood for
the other part of the observations we use the density, as usual. It is clear that the maximum
428
Semiparametric Models
likelihood estimator r, charges all observed values ZI, . . . , Zn, but the term Pe (x 117) leads
to some additional support points as well. In general, these are not equal to values of the
observations.
The score function for e is given by
. .. f K e(xls)pe(xls)d17(S).
le,17(x, y, z) == Ke,17(x) + Ke(Y IZ) == + Ke(Y IZ).
Pe (x 117)
Here Ke (y I z) == a / ae log Pe (y I z) is the score function for e for the conditional density
Pe(Y I z), and Ke,17(x) is the score function for e of the rnixture density pe(x 117).
Paths of the form d17t == (1 + th) dr] (with 17h == O) yield scores
Be ryh(x, z) = Ce ryh(x) + h(z) = f h(s)pe(x I s) d'f/(s) + h(z).
" pe (x 117)
The operator Ce, 17 : L 2 (17) f---+ L 2 (pe (. 117)) is the score operator for the mixture part of the
model. Its Hilbert-space adjoint is given by
C ,ryg(z) = f g(x) pe (x Iz) dp,(x).
The range of B e , 17 is contained in the subset G of L 2 (pe (. Ir]) x 17) consisting of functions
of the form (x, z) f---+ gl (x) + g2 (z) + c. This representation of a function of this type is
unique if both gl and g2 are taken to be mean-zero functions. With P e ,17 the distribution of
the observation (X, Y, Z),
P e ,17(gl E9 g2 E9 c)Be,17 h == Pe,17gI Ce,17 h + 17g2 h + 217 hc == r](C;,17 gI + g2 + 2c)h.
Thus, the adjoint B ,17: G f---+ L 2 (17) of the operator B e ,17: L 2 (r]) f---+ G is given by
B;,17(gl E9g2 E9c) == C;,17 gI +g2 +2c.
Consequently, on the set of mean-zero functions in L 2 (17) we have the identity B ,17Be,17 ==
C ,17Ce,17 + l. Because the operator C ,17Ce,17 is nonnegative definite, the operator B ,17Be,17
is strictly positive definite and hence continuously invertible as an operator of L 2 (17) into
itself. The following lemma gives a condition for continuous invertibility as an operator
on the space C a (Z) of all "a-smooth functions." For aa < a the smallest integer strictly
smaller than a, these consist of the functions h : Z C JRd f---+ JR. whose partial derivatives up
to order aa exist and are bounded and whose aa-order partial derivatives are Lipschitz of
order a - aa. These are Banach spaces relative to the norm, with Dk a differential operator
a k ! . . . a kd / a Z l . . . Z d ,
Ilhll a == max sup IDkh(z)1 v max sup
Ikl<a ZEZ Ikl=ao Zl#Z2 EZ
I Dk (ZI) - D k (Z2)1
IlzI - z21l a - ao
The unit ball of one of these spaces is a good choice for the set H indexing the likelihood
equations if the maps z f---+ peo (x I z) are sufficiently smooth.
25.96 Lemma. Let Z be a bounded, convex subset of JRd and assume that the maps Z f---+
Pa (x I z) are continuously differentiable for each x with partial derivatives a / a Zi pe o (x I z)
25.12 Likelihood Equations
429
satisfying, for all Z, ZI in Z and fixed constants K and a > O,
f a a I I
-po(x Iz) - -po(x Iz) df-L(x) < K Ilz - z W ,
a Zi a Zi
f PO(X Iz) dfJ-(x) < K.
aZi
Then B;O,1]O Beo,r]O : Cf3 (Z) r--+ Cf3 (Z) is continuously invertible for every tJ < a.
Proof. By its strict positive-definiteness in the Hilbert-space sense, the operator B Bo :
£,00 (Z) r--+ £,00 (Z) is certainly one-ta-one in that B Boh == O implies that h == O almost
surely under 170. On reinserting this we find that -h == C Coh == C O == O everywhere.
Thus B Bo is also one-to-one in a pointwise sense. If it can be shown that C Co : C tJ (Z) r--+
CtJ (Z) is compact, then B Bo is onto and continuously invertible, by Lemma 25.93.
It follows from the Lipschitz condition on the partial derivatives that C h (z) is differ-
entiable for every bounded function h : X r--+ JR. and its partial derivatives can be found by
differentiating under the integral sign:
a f a
-C h(z) == h(x) - po(x Iz) df-L(x).
aZi aZi
The two conditions of the lemma imply that this function has Lipschitz norm of order a
bounded by K IIh 1100. Let h n be a uniformly bounded sequence in £,00 (X). Then the p arti al
derivatives of the sequence C hn are uniformly bounded and have uniformly bounded
Lipschitz norms of order ex. Because Z is totally bounded, it follows by a strengthening
of the Arzela- Ascoli theorem that the sequences of p arti al derivatives are precompact with
respect to the Lipschitz norm of order tJ for every tJ < a. Thus there exists a subsequence
along which the p arti al derivatives converge in the Lipschitz norm of order tJ. By the
Arzela-Ascoli theorem there exists a further subsequence such that the functions C hn (z)
converge uniformly to alimit. If both a sequence of functions its elf and their continuous
partial derivatives converge uniformly to limits, then the limit of the functions must have
the limits of the sequences of partial derivatives as its partial derivatives. We conclude that
C hn converges in the II . IIl+f3-norm, whence C : £,oo(X) r--+ CtJ (Z) is compact. Then the
operator C Co is certainly compact as an operator from Cf3 (Z) into its elf. .
Because the efficient information for e is bounded below by the information for e in
a "good" observation (Y, Z), it is typically positive. Then the preceding lemma together
with Lemma 25.92 shows that the derivative \11 0 is continuously invertible as a map from
JR.k x £,oo(Ji) X ffi.k x £,00(1i) for Ji the unit ball of Cf3 (Z). This is useful in the cases that
the dimension of Z is not bigger than 3, for, in view of Example 19.9, we must have that
tJ > d /2 in order that the functions Be,1]h == C e ,1]h EB h form a Donsker class, as required
by Theorem 25.90. Thus a > 1/2,2,3/2 suffice in dimensions 1,2,3, but we need tJ > 2
if Z is of dimension 4.
Sets Z ofhigher dimension can be treated by extending Lemma 25.96 to take into account
higher-order derivatives, or a1ternatively, by not using a cet (Z)-unit ball for Ji. The general
requirements for a class Ji that is the unit ball of a Banach space Jffi are that Ji is 170- Donsker,
that C CoIB c IB, and that C Co : IB r--+ IB is compact. For instance, if pe (x Iz) corresponds
to a linear regression on z, then the functions z r--+ C Coh(z) are of the form z r--+ g(a T z)
430
Semiparametric Models
for functions g with a one-dimensional domain. Then the dimensionality of Z does not
really play an important role, and we can apply similar arguments, under weaker conditions
than required by treating Z as general higher dimensional, with, for instance, IBS equal to
the Banach space consisting of the linear span of the functions z r-+ g(a T z) in Cf (Z) and
H its unit ball.
The second main condition of Theorem 25.92 is that the functions ž(),'1 and Be,'1h form
a Donsker class. Dependent on the kern el pe(x Iz), a variety of methods may be used
to verify this condition. One possibility is to employ smoothness of the kemel in x in
combination with Example 19.9. Ifthe map x 1--* p()(x I z) is appropriately smooth, then so
is the map X 1--* C(), '1h (x). Straightforward differentiation yields
CO''lh(x) = cov x ( h(Z), log PO (x IZ) ) ,
aXi aXi
where for each X the covariance is computed for the random variable Z having the (condi-
tional) density z 1--* p()(x Iz) drJ(z)jp()(x I rJ). Thus, for a given bounded function h,
aJI a:. log p() (x I z) I p() (x Iz) drJ(z)
-C () h ( x ) < Il h ll I .
a ,'1 - 00 J ( I ) d ( )
Xi p() X Z 1] Z
Depending on the function a jaxi log p() (x I z), this leads to a bound on the first derivative of
the function x 1--* C (), '1 h (x ). If X is an interval in , then this is sufficient for applicability of
Example 19.9. If Xis higher dimensional, the we can bound higher-order p arti al derivatives
in a simi1ar manner.
If the main interest is in the estimation of rJ rather than e, then there is also a nontechnical
criterion for the choice of H, because the final result gives the asymptotic distribution of f]h
for every h E H, but not necessarily for h tj. H. Typically, a particular h of interest can be
added to a set H that is chosen for technical reasons without violating the resuits as given
previously. The addition of an infinite set would require additional arguments. Reference
[107] gives more detai1s conceming this example.
Notes
Most of the results in this chapter were obtained during the past 15 years, and the area is
still in development. The monograph by Bickel, Klaassen, Ritov, and Wellner [8] gives
many detailed informatian calculations, and heuristic discussions of methods to construct
estimators. See [77], [101], [102], [113], [122], [145] for a number of other, also more re-
cent, papers. For many applications in survival analysis, counting processes offer a flexible
modeling tool, as shown in Andersen, Borgan, Gill, and Keiding [1], who also treat semi-
parametric model s for survival analysis. The treatment of maximum likelihood estimators
is motivated by (partially unpublished) joint work with Susan Murphy. Apparently, the
present treatment of the Cox model is novel, although proofs using the profile like1ihood
function and martingales go back at least 15 years. In connection with estimating equations
and CAR models we profited from discussions with James Robins, the representation in
section 25.53 going back to [129]. The use of the empiricallikelihood goes back a long
way, in particular in survival analysis. More recently it has gained popularity as a basis
for constructing likelihood ratio based confidenee intervals. Limitations of the information
25.12 Likelihood Equations
431
bounds and the type of asymptotics discussed in this chapter are pointed out in [128]. For
further information concerning this chapter consult recent joumals, both in statistics and
econometrics.
PROBLEMS
1. Suppose that the under1ying distribution of arandom sample of real-valued observations is known
to have mean zero but is otherwise unknown.
(i) Derive a tangent set for the model.
(ii) Find the efficient influence function for estimating 1/1 (P) == P (C) for a fixed set C.
(iii) Find an asymptotically efficient sequence of estimators for 1/1(P).
2. Suppose that the model consists of densities p(x - e) on IR k , where p is a smooth density with
p (x) == p ( - x). Find the efficient inftuence function for estimating e.
3. In the regression model of Example 25.28, assume in addition that e and X are independent. Find
the efficient score function for e.
4. Find a tangent set for the set of mixture distributions f p(x Iz) dF(z) for x t--+ p(x I z) the
uniform distribution on [z, z + 1]. Is the linear span of this set equal to the nonparametric tangent
set?
5. (Neyman-Scott problem) Suppose that a typical observation is a pair (X, Y) of variables that
are conditionally independent and N (Z, e)-distributed given an unobservable variable Z with a
completely unknown distribution rJ on IR. A natural approach to estimating e is to "eliminate"
the unobservable Z by taking the difference X - Y. The maximum likelihood estimator based
on a sample of such differences is Tn == !n- 1 I:7=1 (Xi - Yi)2.
(i) Show that the closed linear span of the tangent set for rJ contains all square-integrable,
mean-zero functions of X + Y.
(ii) Show that Tn is asymptotically efficient.
(iii) Is Tn equal to the semiparametric maximum likelihood estimator?
6. In Example 25.72, calculate the score operator and the information operator for rJ.
7. In Example 25.12, express the density of an observation X in the marginal distributions F and
G of Y and C and
(i) Calculate the score operators for F and G.
(ii) Show that the empirical distribution functions p* and (;* of the Yi and ej are asymptotically
efficient for estimating the marginal distributions F* and G* of Y and C, respectively;
(iii) Prove the asymptotic normality of the estimator for F given by
F(y) == 1 - TI (1 - A{s}),
o:::;s:::;y
A 1 dP*
A (y) == A * A * ;
[O,y] G - F
(iv) Show that this estimator is asymptotically efficient.
8. (Star-shaped distributions) Let F be the collection of all cumulative distribution functions on
[O, 1] such that x t--+ F (x) Ix is nondecreasing. (This is a famous example in which the maximum
likelihood estimator is inconsistent.)
(i) Showthatthereexistsamaximizer P n (over F) of the empirical likelihood F t--+ [17=1 F{Xi},
and show that this satisfies F n (x) -* x F (x) for every x.
(ii) Show that at every F E F there is a convex tangent cone whose closed linear span is the
nonparametric tangent space. What does this mean for efficient estimation of F?
432
Semiparametric Models
9. Show that a U -statistic is an asymptotically efficient estimator for its expectation if the model is
nonparametric.
10. Suppose that the model consists of all probability distributions on the real line that are symmetric.
(i) If the symmetry point is known to be O, find the maximum likelihood estimator relative to
the empiricallikelihood.
(ii) If the symmetry point is unknown, characterize the maximum likelihood estimators relative
to the empiricallikelihood; are they useful?
11. Find the profile likelihood function for the parameter () in the Cox model with censoring discussed
in Section 25.12.1.
12. Let P be the set of all probability distributions on IR with apositive density and let 1/r (P) be the
median of P.
(i) Find the influence function of 1/r .
(ii) Prave that the sample median is asymptotically efficient.
References
[1] Andersen, P.K., Borgan, O., Gill, R.D., and Keiding, N. (1992). Statistical Models Based on
Counting Processes. Springer, Berlin.
[2] Arcones, M.A., and Gine, E. (1993). Limit theorems for U-processes. Annals of Probability
21, 1494-1542.
[3] Bahadur, R.R. (1967). An optimal property of the likelihood ratio statistic. Proceedings of
the Fifth Berkeley Symposium on Mathematical Statistics and Probability (1965/66) I, 13-26.
University of California Press, Berkeley.
[4] Bahadur, R.R. (1971). Some limit theorems in statisties. Conference Board of the Mathematical
Sciences Regional Conference Series in Applied Mathematics 4. Society for Industrial and
Applied Mathematics, Philadelphia.
[5] Bamdoff-Nielsen, O.E., and Hall, P. (1988). On the level-error after Bartlett adjustment of the
likelihood ratio statistic. Biometrika 75, 378-378.
[6] Bauer, H. (1981). Probability Theory and Elements of Measure Theory. Holt, Rinehart, and
Winston, New York.
[7] Bentkus, V., Gotze, F., van Zwet, W.R. (1997). An Edgeworth expansion for symmetric statis-
tics. Annals of Statistics 25, 851-896.
[8] Bickel, P.J., Klaassen, C.A.J., Ritov, Y., and Wellner, J.A. (1993). Efficient and Adaptive
Estimationfor Semiparametric Models. Johns Hopkins University Press, Ba1timore.
[9] Bickel, P.J., and Ghosh, J.K. (1990). A decomposition for the likelihood ratio statistic and the
Bartlett correction-a Bayesian argument. Annals of Statistics 18, 1070-1090.
[10] Bickel, P.J., and Rosenblatt, M. (1973). On some global measures of the deviations of density
function estimates. Annals of Statistics 1, 1071-1095.
[11] Billingsley, P. (1968). Convergence of Probability Measures. John Wiley, New York.
[12] Birge, L. (1983). Approximation dans les espaces metriques et theorie de l'estimation.
Zeitschrift fitr Wahrscheinlichkeitstheorie und Verwandte Gebiete 65, 181-238.
[13] Birge, L. (1997). Estimation of unimodal densities without smoothness assumptions. Annals
of Statistics 25, 970-981.
[14] Birge, L., and Massart, P. (1993). Rates of convergence for minimum contrast estimators.
Probability Theory and Related Fields 97, 113-150.
[15] Birge, L., and Massart, P. (1997). From model selection to adaptive estimation. Festschriftfor
Lucien Le Cam. Springer, New York, 55-87.
[16] Birman, M.S., and Solomjak, M.Z. (1967). Piecewise-polynomial approximation of functions
of the classes W p . Mathematics of the USSR Sbornik73, 295-317.
[17] Brown, L. (1987). Fundamentals of Statistical Exponential Families with Applications in Sta-
tistical Decision Theory . Institute of Mathematical Statisties, California.
[18] Brown, L.D., and Fox, M. (1974). Admissibility of procedure s in two-dimensionallocation
parameter problems. Annals of Statistics 2, 248-266.
[19] Cantelli, F.P. (1933). Sulla determinazione empirica delle leggi di probabilita. Giornale
dell'Istituto Italiano degli Attuari 4,421-424.
433
434
References
[20] Chemoff, H. (1952). A measure of asymptotic efficiency for tests of a hypothesis based on the
sum of observations. Annals of Mathen1,atical Statistics 23, 493-507.
[21] Chemoff, H. (1954). On the distribution of the likelihood ratio statistic. Annals of Mathematical
Statistics 25, 573-578.
[22] Chemoff, H., and Lehmann, E.L. (1954). The use ofmaximum likelihood estimates in X 2 tests
for goodness of fit. Annals of Mathematical Statistics 25, 579-586.
[23] Chow, Y.S., and Teicher, H. (1978). Probability Theory. Springer- Verlag, New York.
[24] Cohn, D.L. (1980). Measure Theory. Birkhauser, Boston.
[25] Copas, J. (1975). On the unimodality of the likelihood for the Cauchy distribution. Biometrika
62,701-704.
[26] Cramer, H. (1938). Sur un nouveau theoreme-limite de la theorie des probabilites. Actualites
Scienti.fiques et Industrielles 736, 5-23.
[27] Cramer, H. (1946). Mathematical Methods ofStatistics. Prineeton University Press, Prineeton.
[28] Csorg6, M. (1983). Quantile Processes with Statistical Applications. CBMS-NSF Regional
Conference Series in Applied Mathematics 42. Society for Industrial and Applied Mathematics
(SlAM), Philadelphia.
[29] Dacunha-Castelle, D., and Duflo, M. (1993). Probabilites et Statistiques, tome II. Masson,
Paris.
[30] Davies, R.B. (1973). Asymptotic inference in stationary Gaussian time-series. Advances in
Applied Probability 4, 469-497.
[31] Dembo, A., and Zeitouni, O. (1993). Large Deviation Techniques and Applications. Jones and
Bart1ett Publishers, Boston.
[32] Deuschel, J.D., and Stroock, D.W. (1989). Large Deviations. Academic Press, New York.
[33] Devroye, L., and Gyorfi, L. (1985). Nonparametric Density Estimation: The L1- View. John
Wiley & Sons, New York.
[34] Diaconis, P., and Freedman, D. (1986). On the consistency of Bayes estimates. Annals of
Statistics 14, 1-26.
[35] Donald, S.G., and Newey, W.K. (1994). Series estimation of semilinear models. Journal of
Multivariate Analysis 50, 30-40.
[36] Donoho, D.L., and Johnstone, I.M. (1994). Idea spatial adaptation by wavelet shrinkage.
Biometrika 81,425-455.
[37] Donoho, D.L., and Liu, R.C. (1991). Geometrizing rates of convergence II, III. Annals of
Statistics 19, 633-701.
[38] Donsker, M.D. (1952). Justification and extension of Doob's heuristic approach to the
Kolmogorov-Smirnov theorems. Annals of Mathematical Statistics 23,277-281.
[39] Doob, J. (1948). Application of the theory of martingales. Le Calcul des Probabilites et ses
Applications. Colloques Internationales du CNRS Paris, 22-28.
[40] Drost, F.C. (1988). Asymptoticsfor Generalized Chi-Square Goodness-of-Fit Tests. CWI tract
48. Centrum voor Wiskunde en Informatica, Amsterdam.
[41] Dudley, R.M. (1976). Probability and Metrics: Convergence of Laws on Metric Spaces. Math-
ematics Institute Lecture Notes Series 45. Aarhus University, Denmark.
[42] Dudley, R.M. (1989). Real Analysis and Probability, Wadsworth, Belmont, California.
[43] Dupač, V., and Hajek, J. (1969). Asymptotic normality of simple linear rank statistics under
alternatives, Annals of Mathematical Statistics II 40, 1992-2017.
[44] Efron, B., and Tibshirani, R.J. (1993). An Introduction to the Bootstrap. Chapman and Hall,
London.
[45] Fahrmeir, L., and Kaufmann, H. (1985). Consistency and asymptotic normality of the maximum
likelihood estimator in generalized linear models. Annals of Statistics 13, 342-368. (Correction:
Annals of Statistics 14, 1643.)
[46] Farrell, R.H. (1972). On the best obtainable asymptotic rate s of convergence in estimation of
a density function at a point. Annals of Mathematical Statistics 43, 170-180.
[47] Fel1er, W. (1971). An Introduction to Probability Theory and Its Applications, vol. II. John
Wiley & Sons, New York.
References
435
[48] Fisher, R.A. (1922). On the mathematical foundations of theoretical statisties. Philosophical
Transactions of the Royal Society of London, Series A 222,309-368.
[49] Fisher, R.A. (1924). The conditions under which X 2 measures the discrepancy between obser-
vations and hypothesis. Journal Royal Statist. Soc. 87,442-450.
[50] Fisher, R.A. (1925). Theory of statistical estimation. Proceedings of the Cambridge Philosoph-
ical Society 22, 700-725.
[51] van de Geer, S.A. (1988). RegressionAnalysisandEmpiricalProcesses. CWITract45. Centrom
voor Wiskunde en Informatica, Amsterdam.
[52] Ghosh, J.K. (1994). Higher Order Asymptotics. Institute of Mathematical Statisties, Hayward.
[53] Gill, R.D. (1989). Non- and semi-parametric maximum likelihood estimators and the von-Mises
method (part I). Scandinavian Journal of Statistics 16, 97-128.
[54] Gill, R.D. (1994). Lectures on survival analysis. Lecture Notes in Mathematics 1581, 115-241.
[55] Gill, R.D., and Johansen, S. (1990). A survey of product-integration with a view towards
application in survival analysis. Annals of Statistics 18, 1501-1555.
[56] Gill, R.D., and van der Vaart, A.W. (1993). Non- and semi-parametric maximum likelihood
estimators and the von Mises method (part II). Scandinavian Journal of Statistics 20, 271-288.
[57] Gine, E., and Zinn, J. (1986). Lectures on the central limit theorem for empirical processes.
Lecture Notes in Mathematics 1221, 50-113.
[58] Gine, E., and Zinn, J. (1990). Bootstrapping general empirical measures. Annals of Probability
18, 851-869.
[59] Glivenko, V. (1933). Sulla determinazione empirica della leggi di probabilita. Giornale
dell'Istituto Italiano degli Attuari 4, 92-99.
[60] Greenwood, P.E., and Nikulin, M.S. (1996). A Guide to Chi-Squared Testing. John Wiley &
Sons, New York.
[61] Groeneboom, P. (1980). Large Deviations and Bahadur Efficiencies. MC tract 118, Centrum
voor Wiskunde en Informatica, Amsterdam.
[62] Groeneboom, P. (1985). Estimating amonotone density. Proceedings of the Berkeley Confer-
ence in Honor of Jerzy Neyman and Jack Kiefer 2, 539-555. Wadsworth, Monterey, California.
[63] Groeneboom, P. (1988). Brownian Motion with a parabolie drift and Airy functions. Probability
Theory and Related Fields 81, 79-109.
[64] Groeneboom, P., Lopuhaa, H.P. (1993). Isotonic estimators of monotone densities and distri-
bution functions: basic facts. Statistica Neerlandica 47, 175-183.
[65] Groeneboom, P., Oosterhoff, J., and Ruymgaart, F. (1979). Large deviation theorems for em-
pirical probability measures. Annals of Probability 7, 553-586.
[66] de Haan, L. (1976). Sample extremes: An elementary introduction. Statistica Neerlandica 30,
161-172.
[67] Hajek, J. (1961). Some extensions of the Wald-Wolfowitz-Noether theorem. Annals of Math-
ematical Statistics 32, 506-523.
[68] Hajek, J. (1968). Asymptotic normality of simple linearrank statistics under alternatives.Annals
of Mathematical Statistics 39,325-346.
[69] Hajek, J. (1970). A characterization of limiting distributions of regular estimates. Zeitschrift
fur Wahrscheinlichkeitstheorie und Verwandte Gebiete 14, 323-330.
[70] Hajek, J. (1972). Local asymptotic minimax and admissibility in estimation. Proceedings of
the Sixth Berkeley Symposium on Mathematical Statistics and Probability 1, 175-194.
[71] Hajek, l., and Šidak, Z. (1967). Theory of Rank Tests. Academic Press, New York.
[72] Hall, P. (1992). The Bootstrap and Edgeworth Expansion. Springer Series in Statistics. Springer-
Verlag, New York.
[73] Hampel, F.R., Ronchetti, E.M., Rousseeuw, PJ., and Stahel, W.A. (1986). Robust Statistics:
the Approach Based on Influence Functions. Wiley, New York.
[74] Helmers, R. (1982). Edgeworth Expansionsfor Linear Combinations of Order Statistics. Math-
ematical Centre Tracts 105. Mathematisch Centrum, Amsterdam.
[75] Hoeffding, W. (1948). A class of statistics with asymptotically normal distribution. Annals of
Mathematical Statistics 19, 293-325.
436
References
[76] Hoffmann-J rgensen, J. (1991). Stochastic Processes on Polish Spaces. Various Publication
Series 39. Aarhus Universitet, Aarhus, Denmark.
[77] Huang, J. (1996). Efficient estimation for the Cox model with interval censoring. Annals of
Statistics 24, 540-568.
[78] Huber, P. (1967). The behaviorofmaximumlikelihoodestimates undernonstandardconditions.
Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability 1,
221-233. University of Califomia Press, Berkeley.
[79] Huber, P. (1974). Robust Statistics. Wiley, New York.
[80] Ibragimov, LA., and Has'minskii, R.Z. (1981). Statistical Estimation: Asymptotic Theory.
Springer-Verlag, New York.
[81] Jagers, P. (1975). Branching Processes with Biological Applications. John Wiley & Sons,
London-New York-Sydney.
[82] Janssen, A., and Mason, D.M. (1990). Nonstandard Rank Tests. Lecture Notes in Statistics 65.
Springer- Verlag, New York.
[83] Jensen, J.L. (1993). A historical sketch and some new results on the improved log likelihood
ratio statistic. Scandinavian Journal of Statistics 20, 1-15.
[84] Kallenberg, W.C.M. (1983). lntermediate efficiency, theory and exampIes. Annals of Statistics
11, 498-504.
[85] Kallenberg, W.C.M., and Ledwina, T. (1987). On IDeal and nonlocal measures of efficiency.
Annals of Statistics 15, 1401-1420.
[86] Kallenberg, W.C.M., Oosterhoff, J., and Schriever, B.F. (1980). The number of classes in
chi-squared goodness-of-fit tests. Journal of the American Statistical Association 80, 959-968.
[87] Kim, J., and Pollard, D. (1990). Cube root asymptotics. Annals of Statistics 18, 191-219.
[88] Kolčinskii, VJ. (1981). On the central limit theorem for empirical measures. Theory of Proba-
bility and Mathematical Statistics 24, 71-82.
[89] Koul, H.L., and Pflug, G.C. (1990). Weakly adaptive estimators in explosive autoregression.
Annals of Statistics 18, 939-960.
[90] Leadbetter, M.R., Lindgren, G., and Rootzen, H. (1983). Extremes and Related Properties of
Random Sequences and Processes. Springer-Verlag, New York.
[91] Le Cam, L. (1953). On some asymptotic properties of maximum likelihood estimates and
related Bayes estimates. University of California Publications in Statistics 1, 277-330.
[92] Le Cam, L. (1960). Locally asymptotically normal families of distributions. University of
California Publications in Statistics 3, 37-98.
[93] Le Cam, L. (1969). TheorieAsymptotique de la Decision Statistique. Les Presses del'Universite
de Montreal, Montreal.
[94] Le Cam, L. (1970). On the assumptions used to prove asymptotic normality of maximum
likelihood estimates. Annals of Mathematical Statistics 41, 802-828.
[95] Le Cam, L. (1972). Limits of experiments. Proceedings of the Sixth Berkeley Symposium on
Mathematical Statistics and Probability 1, 245-261. University of Califomia Press, BerkeIey.
[96] Le Cam, L. (1986). Asymptotic Methods in Statistical Decision Theory. Springer-Verlag,
New York.
[97] Le Cam, L.M., and Yang, G. (1990). Asymptotics in Statistics: Some Basic Concepts. Springer-
Verlag, New York.
[98] Ledoux, M., and Talagrand, M. (1991). Probability in Banach Spaces: Isoperimetry and Pro-
cesses. Springer- Verlag, Berlin.
[99] Lehmann, E.L. (1983). Theory of Point Estimation. Wiley, New York.
[100] Lehmann, E.L. (1991). Testing Statistical Hypotheses, 2nd edition. Wiley, New York.
[101] Levit, B.Y. (1978). Infinite-dimensional informationallowerbounds. Theory ofProbability and
its Applications 23, 388-394.
[102] Mammen, E., and van de Geer, S.A. (1997). Penalized quasi-likelihood estimation in partial
linear models. Annals of Statistics 25, 1014-1035.
[103] Massart, P. (1990). The tight constant in the Dvoretsky-Kiefer-Wolfowitz inequaIity. Annals of
Probability 18, 1269-1283.
References
437
[104] von Mises, R. (1947). On the asymptotic distribution of differentiable statistical functions.
Annals of Mathematical Statistics 18, 309-348.
[105] Murphy, S.A., Rossini, TJ., and van der Vaart, A.W. (1997). MLE in the proportional odds
model. Journal of the American Statistical Association 92, 968-976.
[106] Murphy, S.A., and van der Vaart, A.W. (1997). Semiparametric likelihood ratio inference.
Annals of Statistics 25, 1471-1509.
[107] Murphy, S.A., and van der Vaart, A.W. (1996). Semiparametric mixtures in case-control studies.
[108] Murphy, S.A., and van der Vaart, A.W. (1996). Likelihood ratio inference in the errors-in-
variables model. Journal of Multivariate Analysis 59,81-108.
[109] Noether, G.E. (1955). On a theorem of Pitman. Annals of Mathematical Statistics 25,64-68.
[110] Nussbaum, M. (1996). Asymptotic equivalence of density estimation and Gaussian white noise.
Annals of Statistics 24, 2399-2430.
[111] Ossiander, M. (1987). A central limit theorem under metric entropy with L2 bracketing. Annals
of Probability 15,897-919.
[112] Pearson, K. (1900). On the criterion that a given system of deviations from the probable in
the case of a correlated system of variables is such that it can be reasonably supposed to have
arisen from random sampling. Philosopical Magazine, Series 5 50, 157-175. (Reprinted in:
Karl Pearson's Early Statistical Papers, Cambridge University Press, 1956.)
[113] Pfanzagl, J., and Wefelmeyer, W. (1982). Contributions to a General Asymptotic Statistical
Theory. Lecture Notes in Statistics 13. Springer- Verlag, New York.
[114] Pfanzagl, J., and Wefelmeyer, W. (1985). Asymptotic Expansions for General Statistical Mod-
els. Lecture Notes in Statistics 31. Springer-Verlag, New York.
[115] Pflug, G.C. (1983). The limiting loglikelihood process for discontinuous density families.
Zeitschrift fur Wahrscheinlichkeitstheorie und Verwandte Gebiete 64, 15-35.
[116] Pollard, D. (1982). A central limit theorem for empirical processes. Journal of the Australian
Mathematical Society A 33, 235-248.
[117] Pollard, D. (1984). Convergence ofStochastic Processes. Springer-Verlag, New York.
[118] Pollard, D. (1985). New ways to prove central limit theorems. Econometric Theory 1,295-314.
[119] Pollard, D. (1989). A maximal inequality for sums of independent processes under a bracketing
condition.
[120] Pollard, D. (1990). Empirical Processes: Theory and Applications. NSF-CBMS Regional Con-
ference Series in Probability and Statistics 2. Institute of Mathematical Statistics and American
Statistical Association. Hayward, California.
[121] Prakasa Rao, B.L.S. (1983). Nonparametric Functional Estimation. Academic Press, Orlando.
[122] Qin, J., and Lawless, 1. (1994). Empiricallikelihood and general estimating equations. Annals
of Statistics 22, 300-325.
[123] Rao, C.R. (1973). Linear Statistical Inference and Its Applications. Wiley, New York.
[124] Reed, M., and Simon, B. (1980). Functional Analysis. Academic Press, Orlando.
[125] Reeds, J.A. (1976). On the Dejinition ofvon Mises Functionals. Ph.D. dissertation, Department
of Statisties, Harvard University, Cambridge, MA.
[126] Reeds, J.A. (1985). Asymptotic number of roots of Cauchy location likelihood equations.
Annals of Statistics 13, 775-784.
[127] Revesz, P. (1968). The Laws of Large Numbers. Academic Press, New York.
[128] Robins, J.M., and Ritov, Y. (1997). Towards a curse of dimensionality appropriate (CODA)
asymptotic theory for semi-parametric models. Statistics in Medicine 16, 285-319.
[129] Robins, J.M., and Rotnitzky, A. (1992). Recovery ofinformation and adjustment for dependent
censoring using surrogate markers. In AIDS Epidemiology-Methodological Issues, 297-331,
eds: N. Jewell, K. Dietz, and V. Farewell. Birkhauser, Boston.
[130] Roussas, G.G. (1972). Contiguity of Probability Measures. Cambridge University Press,
Cambridge.
[131] Rubin, H., and Vitale, R.A. (1980). Asymptotic distribution of symmetric statisties. Annals of
Statistics 8, 165-170.
[132] Rudin, W. (1973). Functional Analysis. McGraw-Hill, New York.
438
References
[133] Shiryayev, A.N. (1984). Probability. Springer-Verlag, New York-Berlin.
[134] Shorack, G.R., and Wellner, J.A. (1986). Empirical Processes with Applications to Statisties.
Wiley, New York.
[135] Silverman, B.W. (1986). Density Estimation for Statistics and Data Analysis. Chapman and
Hall, London.
[136] Stigler, S.M. (1974). Linear functions of order statistics with smooth weight functions. Annals
of Statistics 2, 676-693. (Correction: Annals of Statistics 7, 466.)
[137] Stone, C.J. (1990). Large-sample inference for log-spline models. Annals of Statistics, 18,
717-741.
[138] Strasser, H. (1985). Mathematical Theory of Statistics. Walter de Gruyter, Berlin.
[139] van der Vaart, A.W. (1988). Statistical Estimation in Large Parameter Spaces. CWI Tracts 44.
Centrum voor Wiskunde en Informatica, Amsterdam.
[140] van der Vaart, A.W. (1991). On differentiable functionals. Annals of Statistics 19, 178-204.
[141] van der Vaart, A.W. (1991). An asymptotic representation theorem. International Statistical
Review 59,97-121.
[142] van der Vaart, A.W. (1994). Limits of Experiments. Lecture Notes, Yale University.
[143] van der Vaart, A.W. (1995). Efficiency of infinite dimensional M-estimators. Statistica Neer-
landica 49,9-30.
[144] van der Vaart, A.W. (1994). Maximum likelihood estimation with partially censored observa-
tions. Annals of Statistics 22, 1896-1916.
[145] van der Vaart, A. W. (1996). Efficient estimation in semiparametric models. Annals of Statistics
24, 862-878.
[146] van der Vaart, A.W., and Wellner, J.A. (1996). Weak Convergence and Empirical Processes.
Springer, New York.
[147] Vapnik, V.N., and Červonenkis, A.Y. (1971). On the uniform convergence ofrelative frequen-
cies of events to their probabilities. Theory of Probability and Its Applications 16, 264-280.
[148] Wahba, G. (1990). Spline modelsfor observational data. CBMS-NSF Regional Conference Se-
ries in Applied Mathematics 59. Society for Industrial and Applied Mathematics, Philadelphia.
[149] Wald, A. (1943). Test of statistical hypotheses concerning several perameters when the number
of observations is large. Transactions of the American Mathematical Society 54, 426-482.
[150] Wilks, S.S. (1938). The large-sample distribution of the likelihood ratio for testing composite
hypotheses. Annals of Mathematical Statistics 19, 60-62.
[151] van Zwet, W.R. (1984). A Berry-Esseen bound for symmetric statisties. Zeitschrift fur
Wahrscheinlichkeitstheorie und Verwandte Gebiete 66, 425-440.
Index
Ci-Winsorized means, 316
Ci-trimmed means, 316
absolute rank, 181
absolutely
continuous, 85, 268
continuous part, 85
accessible, 150
adaptation, 223
adjoint map, 361
adjoint score operator, 372
antiranks, 184
Assouad' s lemma, 347
asymptotic
eonsistent, 44, 329
differentiable, 106
distribution free, 164
efficient, 64, 367, 387
equicontinuity, 262
influence funetion, 58
of level Ci, 192
linear, 401
lower bound, 108
measurable, 260
relative efficiency, 195
risk, 109
tight, 260
tightness,262
uniformly integrable, 17
bootstrap
empirical distribution, 332
empirieal proeess, 332
parametric, 328, 340
B orel
a-field,256
measurable,256
bounded
Lipschitz metric, 332
in probability, 8
bowl-shaped, 113
bracket,270
braeketing
integral, 270
number, 270
Brownian
bridge, 266
bridge process, 168
motion, 268
Bahadur
efficieney, 203
relative efficiency, 203
slope, 203, 239
Bahadur-Kiefer theorems, 310
Banaeh space, 361
bandwidth, 342
Bartlett correction, 238
Bayes
estimator, 138
risk, 138
Bemstein's inequality, 285
best regular, 115
bilinear map, 295
cadlag, 257
canonieallink funetions, 235
Cartesian product, 257
Cauchy sequence, 255
central
limit theorem, 6
moments, 27
chain role, 298
ehaining argument, 285
charaeteristie funetion, 13
ehi-square distribution, 242
Chibisov-O'Reilly theorem, 273
closed, 255
closure,255
coarsening at random (CAR), 379
eoefficients, 173
compaet, 255,424
eompact differentiability, 297
complete,255
completion, 257
eoneordant, 164
eonditional expeetation, 155
consistent, 3, 149, 193,329
439
440
Index
contiguous, 87
continuity set, 7
continuous,255
continuously differentiable, 297
converge
almost surely, 6
in distribution, 5, 258
in probability, 5
weakly, 305
convergence
in law, 5
of sets, 101, 232
convergence-determining, 18
converges,255
almost surely, 258
in distribution, 258
to alimit experiment, 126
in probability, 258
coordinate projections, 257
Comish-Fisher expansion, 338
covering number, 274
Cox partiallikelihood, 404
Cramer' s
condition, 335
theorem, 208
Cramer-Von Mises statistic, 171, 277, 295
Cramer-Wold device, 16
critical region, 192
cross- validation, 346
cumulant generating function, 205
cumulative hazard function, 180, 300
Efron's percentile method, 327
ellipsoidally symmetric, 84
empirical
bootstrap, 328
difference process, 309
distribution, 42, 269
distribution function, 265
likelihood, 402
process,42,266,269
entropy,274
with bracketing, 270
integral, 274
uniform,274
envelope function, 270
equivalent, 126
equivariant-in-Iaw, 113
errors-in-variables, 83
estimating equations, 41
experiment, 125
exponential
family, 37
inequality, 285
finite approximation, 261
Fisher
information for location, 96
information matrix, 39
Frechet differentiable, 297
full rank, 38
defective distribution function, 9
deficiency distance, 137
degenerate, 167
dense,255
deviance, 234
differentiable, 25, 363, 387
in quadratic mean, 64, 94
differentially equivalent, 106
discretization, 72
discretized, 72
distance function, 255
distribution
free, 174
function, 5
dominated, 127
Donsker, 269
dualspace, 361, 387
Dzhaparidze- Nikulin statistic, 254
Gateaux differentiable, 296
Gaussian chaos, 167
generalized inverse, 254
Gini' s mean difference, 171
Glivenko-Cantelli, 46
Glivenko-Cantelli theorem, 265
good rate function, 209
goodness-of-fit, 248, 277
gradient, 26
Grenander estimator, 356
s-bracket, 270
Edgeworth expansions, 335
efficient
function, 369, 373
influence function, 363
information matrix, 369
score equations, 391
Hajek projection, 157
Hadamard differentiable, 296
Hamming distance, 347
Hellinger
affinity, 211
distance, 211
integral, 213
statistic, 244
Hermite polynomial, 169
Hilbert space, 360
Hoeffding decomposition, 157
Huber estimators, 43
Hungarian embeddings, 269
hypothesis of independence, 247
identifiable, 62
improper, 138
Index
441
influence function, 292
information operator, 372
interior, 255
interquartile range, 317
logit model, 66
loss function, 109, 113
L-statistic, 316
Lagrange multipliers, 214
LAN, 104
large
deviation, 203
deviation principle, 209
law of large numbers, 6
Le Cam' s third lemma, 90
least
concave majorant, 349
favorable, 362
Lebesgue decomposition, 85
Lehmann a1tematives, 180
level, 192
a,215
likelihood
ratio, 86
ratio process, 126
ratio statistic, 228
linear signed rank statistic, 221
link function, 234
Lipschitz, 6
local
criterion functions, 79
empirical measure, 283
limiting power, 194
parameter, 92
parameter space, 101
locally
asymptoticall y
minimax, 120
mixed normal, 131
normal, 104
quadratic, 132
most powerful, 179
scores, 222, 225
signed rank statistics, 183
test, 190
log rank test, 180
M -estimator, 41
Mann- Whitney statistic, 166, 175
marginal
convergence, 11
vectors, 261
weak convergence, 126
Markov's inequality, 10
Marshall's lemma, 357
maximal inequality, 76, 285
maXlmum
likelihood estimator, 42
likelihood statistic, 231
mean
absolute deviation, 280
integrated square error, 344
median
absolute deviation, 310
absolute deviation from the median,
60
test, 178
metric, 255
space, 255
midrank, 173
Mill's ratio, 313
minimax criterion, 113
minimum-chi square estimator, 244
missing at random (MAR), 379
mode, 355
model, 358
mutually contiguous, 87
joint convergence, 11
(k x r) table, 247
Kaplan-Meier estimator, 302
Kendall' s T -statistic, 164
kemel, 161, 342
estimator, 342
method, 341
Kolmogorov-Smimov, 265, 277
Kruskal-Wallis,181
Kullback-Leibler divergence, 56,
62
kurtosis, 27
,jn-consistent,72
natural
level, 190
parameter space, 38
nearly maximize, 45
Nelson-Aalen estimator, 301
Newton-Rhapson,71
noncentral chi -square distribution, 237
noncentrality parameter, 217
nonparametric
maximum likelihood estimator, 403
model, 341, 358
norm, 255
normed space, 255
nuisance parameter, 358
observed
level, 239
significance level, 203
odds,406
offspring distribution, 133
one-step
estimator, 72
method, 71
442
Index
open, 255
ball, 255
operators,361
order statistics, 173
orthocomplement, 361
orthogonal, 85, 153, 361
part, 85
outer probability, 258
right-censored data, 301
robust statistics, 43
quantile, 43
function, 304, 306
transformation, 305
sample
correlation coefficient, 30
path, 260
space, 125
Sanov's theorem, 209
Savage test, 180
score, 173, 221
function, 42, 63, 362
operator, 371
statistic, 231
tests, 220
semimetric, 255
seminorm, 256
semiparametric models, 358
separable, 255
shatter, 275
shrinkage estimator, 119
Siegel-Tukey,191
sign, 181
statistic, 183, 193, 221
sign- function, 43
signed rank
statistic, 181
test, 164
simple linear rank statistics, 173
single-index,359
singular part, 86
size, 192, 215
skewness,29
Skorohod space, 257
slope, 195,218
smoothing method, 342
solid, 237
spectral density, 105
standard (or uniform) Brownian bridge, 266
statistical experiment, 92
Stein' s lemma, 213
stochastic process, 260
Strassen's theorem, 268
strong
approximations, 268
law of large numbers, 6
strongl y
consistent, 149
degenerate, 167
subconvex,113
subgraphs, 275
P-Brownian bridge, 269
p -Donsker, 269
P -Glivenko-Cantelli, 269
pth sample quantile, 43
parametric
bootstrap, 328, 340
models,341
Pearson statistic, 242
percentile
t-method,327
method, 327
perfect, 211
permutation tests, 188
Pitman
efficiency, 201
relative efficiency, 202
polynomial c1asses, 275
pooled sample, 174
posterior distribution, 138
power function, 215
prior distribution, 138
probability integral transformation, 305
probit model, 66
product
integral, 300
limit estimator, 302, 407
profile likelihood, 403
projection, 153
lemma,361
proportional hazards, 180
random
element, 256
vector, 5
randomized statistic, 98, 127
range, 317
rank, 164, 173
correlation coefficient, 184
statistic, 173
Rao-Robson-Nikulin statistic, 250
rate function, 209
rate-adaptive, 346
regular, 1] 5, 365
regularity, 340
relative efficiency, 111, 201
T-topology,209
tangent set, 362
for 17, 369
tangentspace,362
tangentially, 297
test, 191,215
function, 215
Index
443
tight, 8, 260
total
variation, 22
variation distance, 211
totally bounded, 255
two-sample U -statistic, 165
variance stabililizing transformation,
30
VC c1ass, 275
VC index, 275
Volterra equation, 408
von Mises expansion, 292
V -statistic, 172, 295, 303
van der Waerden statistic, 175
Vapnik-Červonenkis c1asses, 274
Wald
statistic, 231
tests, 220
Watson-Roy statistic, 251
weak
convergence, 5
law of large numbers, 6
weighted empirical process, 273
well-separated, 45
Wilcoxon
signed rank statistic, 183, 221
statistic, 175
two-sample statistic, 167
window, 342
U-statistic, 161
unbiased, 226
uniform
covering numbers, 274
entropy integral, 274
Glivenko-Cantelli, 145
norm,257
uniformly
most powerful test, 216
tight, 8
unimodal, 355
uprank, 173
Z -estimators, 41