Chapter 3 Probability theory

3.1 Background material

Theorem 3.1 If \(P\) is a probability function, then

1. \(P\left(A\right)=\sum_{i=1}^{\infty}P\left(A\cap C_{i}\right)\) for any partition \(C_{1},C_{2},\ldots\);
2. \(P\left(\cup_{i=1}^{\infty}A_{i}\right)\leq\sum_{i=1}^{\infty}P\left(A_{i}\right)\) for any sets \(A_{1},A_{2},\ldots\).

Theorem 3.2 (Bayes’ Rule) Let \(A_{1},A_{2},\ldots\) be a partition of the sample space \(\Omega\), and let \(B\) be any set. Then, for each \(i=1,2,\ldots\), \[ P\left(A_{i}|B\right)=\frac{P\left(B|A_{i}\right)P\left(A_{i}\right)}{\sum_{j=1}^{\infty}P\left(B|A_{j}\right)P\left(A_{j}\right)}. \]

Proof. From the definition of conditional probability, we have \[\begin{align*} P\left(A_{i}|B\right) & =\frac{P\left(A_{i}\cap B\right)}{P\left(B\right)} \\ & =\frac{P\left(B|A_{i}\right)P\left(A_{i}\right)}{P\left(B\right)}\tag{conditional probability} \\ & =\frac{P\left(B|A_{i}\right)P\left(A_{i}\right)}{\sum_{j=1}^{\infty}P\left(B\cap A_{j}\right)}\tag{the $A_{j}$ partition $\Omega$} \\ & =\frac{P\left(B|A_{i}\right)P\left(A_{i}\right)}{\sum_{j=1}^{\infty}P\left(B|A_{j}\right)P\left(A_{j}\right)}.\tag{conditional probability} \end{align*}\]

Definition 3.1 The cumulative distribution function or cdf of a random variable \(X\), denoted by \(F_{X}\left(x\right)\), is defined by \(F_{X}\left(x\right)=P_{X}\left(X\leq x\right)\), for all \(x\).

Theorem 3.3 The function \(F\left(x\right)\) is a cdf if and only if the following three conditions hold:

1. \(\lim_{x\rightarrow-\infty}F\left(x\right)=0\) and \(\lim_{x\rightarrow\infty}F\left(x\right)=1\).
2. \(F\left(x\right)\) is a nondecreasing function of \(x\).
3. \(F\left(x\right)\) is right-continuous; that is, for every number \(x_{0}\), \(\lim_{x\downarrow x_{0}}F\left(x\right)=F\left(x_{0}\right)\).

(This is Theorem 1.5.3 from Casella and Berger (2002).)

Proof. PROOF GOES HERE

3.2 Transformations and expectations

3.2.1 Distributions of functions of a random variable

When transformations are made, it is important to keep track of the sample spaces of the random variables; otherwise, much confusion can arise. When the transformation is from \(X\) to \(Y=g\left(X\right)\), it is most convenient to use

\[\begin{equation} \mathcal{X}=\left\{ x:f_{X}\left(x\right)>0\right\} \quad\text{and}\quad\mathcal{Y}=\left\{ y:y=g\left(x\right)\text{ for some }x\in\mathcal{X}\right\}. \tag{3.1} \end{equation}\]

Theorem 3.4 Let \(X\) have cdf \(F_{X}\left(x\right)\), let \(Y=g\left(X\right)\), and let \(\mathcal{X}\) and \(\mathcal{Y}\) be defined as in Equation (3.1).

1. If \(g\) is an increasing function on \(\mathcal{X}\), \(F_{Y}\left(y\right)=F_{X}\left(g^{-1}\left(y\right)\right)\) for \(y\in\mathcal{Y}\).
2. If \(g\) is a decreasing function on \(\mathcal{X}\) and \(X\) is a continuous random variable, \(F_{Y}\left(y\right)=1-F_{X}\left(g^{-1}\left(y\right)\right)\) for \(y\in\mathcal{Y}\).

(This is Theorem 2.1.3 from Casella and Berger (2002).)

Proof. \(g\) is a monotone function, i.e., it maps each \(x\) to a single \(y\), and each \(y\) comes from at most one \(x\). In this case that \(g\) is increasing, we have

\[\begin{align*} \left\{ x\in\mathcal{X}:g\left(x\right)\leq y\right\} & =\left\{ x\in\mathcal{X}:g^{-1}\left(g\left(x\right)\right)\leq g^{-1}\left(y\right)\right\} \\ & =\left\{ x\in\mathcal{X}:x\leq g^{-1}\left(y\right)\right\}, \end{align*}\]

and the cdf of \(Y\) is

\[\begin{align*} F_{Y}\left(y\right) & =P\left(\left\{ Y\leq y\right\} \right) \\ & =P\left(\left\{ g\left(X\right)\leq y\right\} \right) \\ & =P\left(\left\{ x\in\mathcal{X}:g\left(x\right)\leq y\right\} \right) \\ & =P\left(\left\{ x\in\mathcal{X}:x\leq g^{-1}\left(y\right)\right\} \right) \\ & =\int_{-\infty}^{g^{-1}\left(y\right)}f_{X}\left(x\right)\dif x \\ & =F_{X}\left(g^{-1}\left(y\right)\right). \end{align*}\]

In the case that \(g\) is decreasing, we have

\[\begin{align*} \left\{ x\in\mathcal{X}:g\left(x\right)\leq y\right\} & =\left\{ x\in\mathcal{X}:g^{-1}\left(g\left(x\right)\right)\geq g^{-1}\left(y\right)\right\} \\ & =\left\{ x\in\mathcal{X}:x\geq g^{-1}\left(y\right)\right\}, \end{align*}\]

and the cdf of \(Y\) is

\[\begin{align*} F_{Y}\left(y\right) & =P\left(\left\{ Y\leq y\right\} \right) \\ & =P\left(\left\{ g\left(X\right)\leq y\right\} \right) \\ & =P\left(\left\{ x\in\mathcal{X}:g\left(x\right)\leq y\right\} \right) \\ & =P\left(\left\{ x\in\mathcal{X}:x\geq g^{-1}\left(y\right)\right\} \right) \\ & =\int_{g^{-1}\left(y\right)}^{\infty}f_{X}\left(x\right)\dif x \\ & =1-F_{X}\left(g^{-1}\left(y\right)\right).\tag{continuity of $X$} \end{align*}\]

Theorem 3.5 Let \(X\) have pdf \(f_{X}\left(x\right)\) and let \(Y=g\left(X\right)\), where \(g\) is a monotone function. Let \(\mathcal{X}=\left\{ x:f_{X}\left(x\right)>0\right\}\) and let \(\mathcal{Y}=\left\{ y:y=g\left(x\right),x\in\mathcal{X}\right\}\). Suppose that \(f_{X}\left(x\right)\) is continuous on \(\mathcal{X}\) and that \(g^{-1}\left(y\right)\) has a continuous derivative on \(\mathcal{Y}\). Then the pdf of \(Y\) is given by

\[ f_{Y}\left(y\right)= \begin{cases} f_{X}\left(g^{-1}\left(y\right)\right)\left|\dfrac{\dif}{\dif y}g^{-1}\left(y\right)\right|, & y\in\mathcal{Y}\\ 0, & \text{otherwise} \end{cases}. \]

(This is Theorem 2.1.5 from Casella and Berger (2002).)

Proof. From Theorem 3.4 and applying the chain rule, we have

\[ f_{Y}\left(y\right)= \dfrac{\dif}{\dif y}F_{Y}\left(y\right)= f_{X}\left(g^{-1}\left(y\right)\right)\dfrac{\dif}{\dif y}g^{-1}\left(y\right) \]

in the case that g is increasing and

\[ f_{Y}\left(y\right)= \dfrac{\dif}{\dif y}F_{Y}\left(y\right)= 0-f_{X}\left(g^{-1}\left(y\right)\right)\dfrac{\dif}{\dif y}g^{-1}\left(y\right)= -f_{X}\left(g^{-1}\left(y\right)\right)\dfrac{\dif}{\dif y}g^{-1}\left(y\right) \]

in the case that \(g\) is decreasing, which can be expressed concisely as in the theorem.

We will look at \(F_{X}^{-1}\), the inverse of the cdf \(F_{X}\). If \(F_{X}\) is strictly increasing, then \(F_{X}^{-1}\) is well defined by

\[\begin{equation} F_{X}^{-1}\left(y\right)=x\implies F_{X}\left(x\right)=y. \tag{3.2} \end{equation}\]

However, if \(F_{X}\) is constant on some interval, then \(F_{X}^{-1}\) is not well defined by Equation (3.2). Any \(x\) satisfying \(x_{1}\leq x\leq x_{2}\) satisfies \(F_{X}\left(x\right)=y\). This problem is avoided by defining \(F_{X}^{-1}\left(y\right)\) for \(0<y<1\) by

\[ F_{X}^{-1}\left(y\right)=\inf\left\{ x:F_{X}\left(x\right)\geq y\right\}, \] a definition that agrees with Equation (3.2) when \(F_{X}\) is nonconstant and provides an \(F_{X}^{-1}\) that is single-valued even when \(F_{X}\) is not strictly increasing. Using this definition, for some interval \(\left(x_{1},x_{2}\right)\) on which \(F_{X}\) is constant, we have \(F_{X}^{-1}\left(y\right)=x_{1}\). At the endpoints of the range of \(y\), \(F_{X}^{-1}\left(y\right)\) can also be defined. \(F_{X}^{-1}\left(1\right)=\infty\) if \(F_{X}\left(x\right)<1\) for all \(x\) and, for any \(F_{X}\), \(F_{X}^{-1}\left(0\right)=-\infty\).

Theorem 3.6 (Probability integral transformation) Let \(X\) have continuous cdf \(F_{X}\left(x\right)\) and define the random variable \(Y\) as \(Y=F_{X}\left(X\right)\). Then \(Y\) is uniformly distributed on \(\left(0,1\right)\), that is, \(P\left(\left\{ Y\leq y\right\} \right)=y\), \(0<y<1\).

(This is Theorem 2.1.10 from Casella and Berger (2002); the following proof is given there.)

Proof. For \(Y=F_{X}\left(X\right)\) we have, for \(0<y<1\),

\[\begin{align*} P\left(\left\{ Y\leq y\right\} \right) & =P\left(\left\{ F_{X}\left(X\right)\leq y\right\} \right) \\ & =P\left(\left\{ F_{X}^{-1}\left[F_{X}\left(X\right)\right]\leq F_{X}^{-1}\left(y\right)\right\} \right)\tag{$F_{X}^{-1}$ is increasing} \\ & =P\left(\left\{ X\leq F_{X}^{-1}\left(y\right)\right\} \right)\tag{see paragraph below} \\ & =F_{X}\left(F_{X}^{-1}\left(y\right)\right)\tag{definition of $F_{X}$} \\ & =y.\tag{continuity of $F_{X}$} \end{align*}\]

At the endpoints we have \(P\left(\left\{ Y\leq y\right\} \right)=1\) for \(y\geq 1\) and \(P\left(\left\{ Y\leq y\right\} \right)=0\) for \(y\leq 0\), showing that \(Y\) has a uniform distribution.

The reasoning behind the equality

\[ P\left(\left\{ F_{X}^{-1}\left(F_{X}\left(X\right)\right)\leq F_{X}^{-1}\left(y\right)\right\} \right)=P\left(\left\{ X\leq F_{X}^{-1}\left(y\right)\right\} \right) \]

is somewhat subtle and deserves additional attention. If \(F_{X}\) is strictly increasing, then it is true that \(F_{X}^{-1}\left(F_{X}\left(x\right)\right)=x\). However, if \(F_{X}\) is flat, it may be that \(F_{X}^{-1}\left(F_{X}\left(x\right)\right)\neq x\). Suppose \(F_{X}\) contains an interval \(\left(x_{1},x_{2}\right)\) on which \(F_{X}\) is constant, and let \(x\in\left[x_{1},x_{2}\right]\). Then \(F_{X}^{-1}\left(F_{X}\left(x\right)\right)=x_{1}\) for any \(x\) in this interval. Even in this case, though, the probability equality holds, since \(P\left(\left\{ X\leq x\right\} \right)=P\left(\left\{ X\leq x_{1}\right\} \right)\) for any \(x\in\left[x_{1},x_{2}\right]\). The flat cdf denotes a region of \(0\) probability \((P\left(\left\{ x_{1}<X\leq x\right\} \right)=F_{X}\left(x\right)-F_{X}\left(x_{1}\right)=0)\).

3.2.2 Expected values

Theorem 3.7 Let \(X\) be a random variable and let \(a\), \(b\), and \(c\) be constants. Then for any functions \(g_{1}\left(x\right)\) and \(g_{2}\left(x\right)\) whose expectations exist,

1. \(\E\left[ag_{1}\left(X\right)+bg_{2}\left(X\right)+c\right]=a\E\left[g_{1}\left(X\right)\right]+b\E\left[g_{2}\left(X\right)\right]+c\).
2. If \(g_{1}\left(x\right)\geq 0\) for all \(x\), then \(\E\left[g_{1}\left(X\right)\right]\geq 0\).
3. If \(g_{1}\left(x\right)\geq g_{2}\left(x\right)\) for all \(x\), then \(\E\left[g_{1}\left(X\right)\right]\geq\E\left[g_{2}\left(X\right)\right]\).
4. If \(a\leq g_{1}\left(x\right)\leq b\) for all \(x\), then \(a\leq\E\left[g_{1}\left(X\right)\right]\leq b\).

(This is Theorem 2.2.5 from Casella and Berger (2002); the following proof is given there.)

Proof. PROOF GOES HERE

3.2.3 Moments and moment generating functions

Definition 3.2 For each integer \(n\), the \(n\text{th}\) moment of \(X\) or \((F_{X}\left(x\right))\), \(\mu'_{n}\), is

\[ \mu'_{n}=\E\left[X^{n}\right]. \]

The \(n\text{th}\) central moment of \(X\), \(\mu_{n}\), is

\[ \mu_{n}=\E\left[\left(X-\mu\right)^{n}\right], \]

where \(\mu=\mu'_{1}=\E\left[X\right]\).

Definition 3.3 Let \(X\) be a random variable with cdf \(F_{X}\). The moment generating function (mgf) of \(X\) (or \(F_{X}\)), denoted by \(M_{X}\left(t\right)\), is

\[ M_{X}\left(t\right)=\E\left[\mathrm{e}^{tX}\right], \]

provided that the expectation exists for \(t\) in some neighborhood of \(0\). That is, there is an \(h>0\) such that, for all \(t\) in \(-h<t<h\), \(\E\left[\mathrm{e}^{tX}\right]\) exists. If the expectation does not exist in a neighborhood of \(0\), we say that the moment generating function does not exist.

More explicitly, we can write the mgf of \(X\) as

\[ M_{X}\left(t\right)=\int_{-\infty}^{\infty}\mathrm{e}^{tx}f_{X}\left(x\right)\dif x \] if \(X\) is continuous, or

\[ M_{X}\left(t\right)=\sum_{x}\mathrm{e}^{tx}P\left(\left\{ X=x\right\} \right) \]

if \(X\) is discrete.

Theorem 3.8 If \(X\) has mgf \(M_{X}\left(t\right)\), then

\[ \E\left[X^{n}\right]=M_{X}^{\left(n\right)}\left(0\right), \]

where we define

\[ M_{X}^{\left(n\right)}\left(0\right)=\frac{\dif^{n}}{\dif t^{n}}M_{X}\left(t\right)\Bigr\vert_{t=0}. \]

That is, the \(n\text{th}\) moment is equal to the \(n\text{th}\) derivative of \(M_{X}\left(t\right)\) evaluated at \(t=0\).

(This is Theorem 2.3.7 from Casella and Berger (2002); the following proof is given there.)

Proof. Assuming that we can differentiate under the integral sign, we have

\[\begin{align*} \frac{\dif}{\dif t}M_{X}\left(t\right) & =\frac{\dif}{\dif t}\int_{-\infty}^{\infty}\mathrm{e}^{tx}f_{X}\left(x\right)\dif x \\ & =\int_{-\infty}^{\infty}\left(\frac{\dif}{\dif t}\mathrm{e}^{tx}\right)f_{X}\left(x\right)\dif x \\ & =\int_{-\infty}^{\infty}\left(x\mathrm{e}^{tx}\right)f_{X}\left(x\right)\dif x \\ & =\E\left[X\mathrm{e}^{tX}\right]. \end{align*}\]

Thus,

\[ \frac{\dif}{\dif t}M_{X}\left(t\right)\Bigr\vert_{t=0}=\E\left[X\mathrm{e}^{tX}\right]\Big\vert_{t=0}=\E\left[X\mathrm{e}^{0}\right]=\E\left[X\right]. \]

Noting that

\[ \frac{\dif^{n}}{\dif t^{n}}\mathrm{e}^{tx}=\frac{\dif^{n-1}}{\dif t^{n-1}}\left[\frac{\dif}{\dif t}\mathrm{e}^{tx}\right]=\frac{\dif^{n-2}}{\dif t^{n-2}}\left[\frac{\dif}{\dif t}x\mathrm{e}^{tx}\right]=\frac{\dif^{n-2}}{\dif t^{n-2}}x^{2}\mathrm{e}^{tx}=\frac{\dif}{\dif t}x^{n-1}\mathrm{e}^{tx}=x^{n}\mathrm{e}^{tx}, \]

we can establish that

\[\begin{align*} \frac{\dif^{n}}{\dif t^{n}}M_{X}\left(t\right)\Bigr\vert_{t=0} & =\frac{\dif^{n}}{\dif t^{n}}\int_{-\infty}^{\infty}\mathrm{e}^{tx}f_{X}\left(x\right)\dif x\Big\vert_{t=0} \\ & =\int_{-\infty}^{\infty}\left(\frac{\dif^{n}}{\dif t^{n}}\mathrm{e}^{tx}\right)f_{X}\left(x\right)\dif x\Big\vert_{t=0} \\ & =\int_{-\infty}^{\infty}\left(x^{n}\mathrm{e}^{tx}\right)f_{X}\left(x\right)\dif x\Big\vert_{t=0} \\ & =\E\left[X^{n}\mathrm{e}^{tX}\right]\Big\vert_{t=0} \\ & =\E\left[X^{n}\right]. \end{align*}\]

Lemma 3.1 Let \(a_{1},a_{2},\ldots\) be a sequence of numbers converging to \(a\), that is, \(\lim_{n\rightarrow\infty}a_{n}=a\). Then

\[ \lim_{n\rightarrow\infty}\left(1+\frac{a_{n}}{n}\right)^{n}=\mathrm{e}^{a}. \]

(This is Lemma 2.3.14 from Casella and Berger (2002).)

Theorem 3.9 For any constants \(a\) and \(b\), the mgf of the random variable \(aX+b\) is given by

\[ M_{aX+b}\left(t\right)=\mathrm{e}^{bt}M_{X}\left(at\right). \]

(This is Theorem 2.3.15 from Casella and Berger (2002).; the following proof is given there.)

Proof. By definition,

\[ M_{aX+b}\left(t\right)= \E\left[\mathrm{e}^{\left(aX+b\right)t}\right]= \E\left[\mathrm{e}^{\left(aX\right)t}\mathrm{e}^{bt}\right]= \mathrm{e}^{bt}\E\left[\mathrm{e}^{\left(at\right)X}\right]= \mathrm{e}^{bt}M_{X}\left(at\right), \]

proving the theorem.

3.3 Multiple random variables

3.3.1 Conditional distributions and independence

Definition 3.4 Let \(\left(X,Y\right)\) be a bivariate random vector with joint pdf or pmf \(f\left(x,y\right)\) and marginal pdfs or pmfs \(f_{X}\left(x\right)\) and \(f_{Y}\left(y\right)\). Then \(X\) and \(Y\) are called independent random variables if, for every \(x\in\mathbb{R}\) and \(y\in\mathbb{R}\), \(f\left(x,y\right)=f_{X}\left(x\right)f_{Y}\left(y\right)\).

Theorem 3.10 Let \(X\) and \(Y\) be independent random variables.

1. For any \(\mathcal{A}\subset\mathbb{R}\) and \(\mathcal{B}\subset\mathbb{R}\), \(P\left(\left\{ X\in \mathcal{A}\right\} \cap\left\{ Y\in \mathcal{B}\right\} \right)=P\left(\left\{ X\in \mathcal{A}\right\} \right)P\left(\left\{ Y\in \mathcal{B}\right\} \right)\); that is, the events \(\left\{ X\in \mathcal{A}\right\}\) and \(\left\{ Y\in \mathcal{B}\right\}\) are independent events.
2. Let \(g\left(x\right)\) be a function only of \(x\) and \(h\left(y\right)\) be a function only of \(y\). Then

\[ \E\left[g\left(X\right)h\left(Y\right)\right]=\E\left[g\left(X\right)\right]\E\left[h\left(Y\right)\right]. \]

(This is Theorem 4.2.10 from Casella and Berger (2002); the following proof is given there.)

Proof. For continuous random variables, part (2) is proved by noting that

\[\begin{align*} \E\left[g\left(X\right)h\left(Y\right)\right] & =\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}g\left(x\right)h\left(y\right)f\left(x,y\right)\dif x\dif y \\ & =\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}g\left(x\right)h\left(y\right)f_{X}\left(x\right)f_{Y}\left(y\right)\dif x\dif y\tag{independence} \\ & =\int_{-\infty}^{\infty}h\left(y\right)f_{Y}\left(y\right)\int_{-\infty}^{\infty}g\left(x\right)f_{X}\left(x\right)\dif x\dif y \\ & =\left(\int_{-\infty}^{\infty}g\left(x\right)f_{X}\left(x\right)\dif x\right)\left(\int_{-\infty}^{\infty}h\left(y\right)f_{Y}\left(y\right)\dif y\right) \\ & =\E\left[g\left(X\right)\right]\E\left[h\left(Y\right)\right]. \end{align*}\]

The result for discrete random variables is proved by replacing integrals by sums. Part (1) can be proved by series of steps similar to those above or by the following argument. Let \(g\left(x\right)\) be the indicator function of the set \(\mathcal{A}\). Let \(h\left(y\right)\) be the indicator function of the set \(\mathcal{B}\). Note that \(g\left(x\right)h\left(y\right)\) is the indicator function of the set \(\mathcal{C}\subset\mathbb{R}^{2}\) defined by \(\mathcal{C}=\left\{ \left(x,y\right):x\in \mathcal{A},y\in \mathcal{B}\right\}\). Thus using the expectation equality just proved, we have

\[\begin{align*} P\left(\left\{ X\in A\right\} \cap\left\{ Y\in B\right\} \right) & =P\left(\left\{ \left(X,Y\right)\in C\right\} \right) \\ & =\E\left[g\left(X\right)h\left(Y\right)\right] \\ & =\E\left[g\left(X\right)\right]\E\left[h\left(Y\right)\right] \\ & =P\left(\left\{ X\in A\right\} \right)P\left(\left\{ Y\in B\right\} \right). \end{align*}\]

Theorem 3.11 Let \(X\) and \(Y\) be independent random variables with moment generating functions \(M_{X}\left(t\right)\) and \(M_{Y}\left(t\right)\). Then the moment generating function of the random variable \(Z=X+Y\) is given by \(M_{Z}\left(t\right)=M_{X}\left(t\right)M_{Y}\left(t\right)\).

(This is Theorem 4.2.12 from Casella and Berger (2002); the following proof is given there.)

Proof. Using the definition of the mgf and Theorem 3.10, we have

\[ M_{Z}\left(t\right)= \E\left[\mathrm{e}^{tZ}\right]= \E\left[\mathrm{e}^{t\left(X+Y\right)}\right]= \E\left[\mathrm{e}^{tX}\mathrm{e}^{tY}\right]= \E\left[\mathrm{e}^{tX}\right]\E\left[\mathrm{e}^{tY}\right]= M_{X}\left(t\right)M_{Y}\left(t\right). \]

3.3.2 Covariance and correlation

Definition 3.5 The covariance of \(X\) and \(Y\) is the number defined by

\[ \Cov\left(X,Y\right)=\E\left[\left(X-\mu_{X}\right)\left(Y-\mu_{Y}\right)\right]. \]

Definition 3.6 The correlation of \(X\) and \(Y\) is the number defined by

\[ \rho_{XY}=\frac{\Cov\left(X,Y\right)}{\sigma_{X}\sigma_{Y}}. \]

The value \(\rho_{XY}\) is also called the correlation coefficient.

Theorem 3.12 If \(X\) and \(Y\) are independent random variables, then \(\Cov\left(X,Y\right)=0\) and \(\rho_{XY}=0\).

(This is Theorem 4.5.5 from Casella and Berger (2002); the following proof is given there.)

Proof. Since \(X\) and \(Y\) are independent, we have \(\E\left[XY\right]=\E\left[X\right]\E\left[Y\right]\). Thus

\[\begin{align*} \E\left[\left(X-\mu_{X}\right)\left(Y-\mu_{Y}\right)\right] & =\E\left[XY-\mu_{Y}X-\mu_{X}Y+\mu_{X}\mu_{Y}\right] \\ & =\E\left[XY\right]-\mu_{Y}\E\left[X\right]-\mu_{X}\E\left[Y\right]+\mu_{X}\mu_{Y} \\ & =\E\left[X\right]\E\left[Y\right]-\E\left[Y\right]\E\left[X\right]-\E\left[X\right]\E\left[Y\right]+\E\left[X\right]\E\left[Y\right] \\ & =0 \end{align*}\]

and

\[ \rho_{XY}=\frac{\Cov\left(X,Y\right)}{\sigma_{X}\sigma_{Y}}=\frac{0}{\sigma_{X}\sigma_{Y}}=0. \]

Theorem 3.13 If \(X\) and \(Y\) are any two random variables and \(a\) and \(b\) are any two constants, then

\[ \Var\left(aX+bY\right)=a^{2}\Var\left(X\right)+b^{2}\Var\left(Y\right)+2ab\Cov\left(X,Y\right). \]

If \(X\) and \(Y\) are independent random variables, then

\[ \Var\left(aX+bY\right)=a^{2}\Var\left(X\right)+b^{2}\Var\left(Y\right). \]

(This is Theorem 4.5.6 from Casella and Berger (2002); the following proof is given there.)

Proof. We have

\[\begin{align*} \Var\left(aX+bY\right) & =\E\left[\left(\left(aX+bY\right)-\E\left[aX+bY\right]\right)^{2}\right] \\ & =\E\left[\left(aX+bY-a\E\left[X\right]-b\E\left[Y\right]\right)^{2}\right] \\ & =\E\left[\left(aX+bY-a\mu_{X}-b\mu_{Y}\right)^{2}\right] \\ & =\E\left[\left(a\left(X-\mu_{X}\right)+b\left(Y-\mu_{Y}\right)\right)^{2}\right] \\ & =\E\left[\left(a\left(X-\mu_{X}\right)\right)^{2}-2ab\left(X-\mu_{X}\right)\left(Y-\mu_{Y}\right)+\left(b\left(Y-\mu_{Y}\right)\right)^{2}\right] \\ & =\E\left[a^{2}\left(X-\mu_{X}\right)^{2}\right]-\E\left[2ab\left(X-\mu_{X}\right)\left(Y-\mu_{Y}\right)\right]+\E\left[b^{2}\left(Y-\mu_{Y}\right)^{2}\right] \\ & =a^{2}\E\left[\left(X-\mu_{X}\right)^{2}\right]-2ab\E\left[\left(X-\mu_{X}\right)\left(Y-\mu_{Y}\right)\right]+b^{2}\E\left[\left(Y-\mu_{Y}\right)^{2}\right] \\ & =a^{2}\Var\left(X\right)+b^{2}\Var\left(Y\right)-2ab\Cov\left(X,Y\right). \end{align*}\] If \(X\) and \(Y\) are independent, it follows from Theorem 3.12 that \(\Cov\left(X,Y\right)=0\) and the second equality is immediate from the first.

Theorem 3.14 For any random variables \(X\) and \(Y\),

1. \(-1\leq\rho_{XY}\leq 1\).
2. \(\left|\rho_{XY}\right|=1\) if and only if there exist numbers \(a\neq 0\) and \(b\) such that \(P\left(\left\{ Y=aX+b\right\} \right)=1\). If \(\rho_{XY}=1\), then \(a>0\), and if \(\rho_{XY}=-1\), then \(a<0\).

(This is Theorem 4.5.7 from Casella and Berger (2002); the following proof is given there.)

Proof. PROOF GOES HERE

3.3.3 Multivariate distributions

Theorem 3.15 Let \(X_{1},\ldots,X_{n}\) be mutually independent random variables with mgfs \(M_{X_{1}}\left(t\right),\ldots M_{X_{n}}\left(t\right)\). Let \(Z=X_{1}+\cdots+X_{n}\). Then the mgf of \(Z\) is

\[ M_{Z}\left(t\right)=M_{X_{1}}\left(t\right)\cdot\cdots\cdot M_{X_{n}}\left(t\right). \]

In particular, if \(X_{1},\ldots,X_{n}\) all have the same distribution with mgf \(M_{X}\left(t\right)\), then \(M_{Z}\left(t\right)=\left(M_{X}\left(t\right)\right)^{n}\).

(This is Theorem 4.6.7 from Casella and Berger (2002), which is a generalization of Theorem 3.11).

3.3.4 Inequalities

Lemma 3.2 (Young’s Inequality) Let \(a\) and \(b\) be any positive numbers, and let \(p\) and \(q\) be any positive numbers (necessarily greater than 1) satisfying

\[ \frac{1}{p}+\frac{1}{q}=1. \]

Then

\[ \frac{1}{p}a^{p}+\frac{1}{q}b^{q}\geq ab \]

with equality if and only if \(a^{p}=b^{q}\).

(This is Lemma 4.7.1 from Casella and Berger (2002); the following proof is given there).

Proof. Fix \(b\), and consider the function

\[ g\left(a\right)=\frac{1}{p}a^{p}+\frac{1}{q}b^{q}-ab. \]

To minimize \(g\left(a\right)\), differentiate and set equal to \(0\):

\[ \frac{\dif}{\dif a}g\left(a\right)=0\implies a^{p-1}-b=0\implies b=a^{p-1}. \]

We will evaluate the second derivative of \(g\) with respect to \(a\) at

\[ b=a^{p-1}\implies b^{1/\left(p-1\right)}=\left(a^{p-1}\right)^{1/\left(p-1\right)}\implies a=b^{1/\left(p-1\right)} \]

to verify that this is a minimum.

\[\begin{align*} \frac{\dif^{2}}{\dif a^{2}}\left[g\left(a\right)\right|_{a=b^{1/\left(p-1\right)}} & =\frac{\dif}{\dif a}\left[a^{p-1}-b\right|_{a=b^{1/\left(p-1\right)}} \\ & =\left[\left(p-1\right)a^{p-2}\right|_{a=b^{1/\left(p-1\right)}} \\ & =\left(p-1\right)\left(b^{1/\left(p-1\right)}\right)^{p-2} \\ & =\left(p-1\right)b^{\left(p-2\right)/\left(p-1\right)} \\ & =\left(p-1\right)b^{\left(p-1-1\right)/\left(p-1\right)} \\ & =\left(p-1\right)b^{\left[\left(p-1\right)/\left(p-1\right)\right]-1/\left(p-1\right)} \\ & =\left(p-1\right)b^{1-1/\left(p-1\right)} \end{align*}\]

We have \(p>1\) and \(b>0\), so that \(p-1>0\) and \(b^{1-1/\left(p-1\right)}>0\). It follows that \(b=a^{p-1}\) is a minimum. We have

\[ \frac{1}{p}+\frac{1}{q}=1\implies\frac{1}{q}=1-\frac{1}{p}=\frac{p-1}{p}\implies q\left(p-1\right)=p, \]

so that the value of \(g\left(a\right)\) at the minimum is

\[ \frac{1}{p}a^{p}+\frac{1}{q}\left(a^{p-1}\right)^{q}-aa^{p-1}=\frac{1}{p}a^{p}+\frac{1}{q}a^{p}-a^{p}=a^{p}\left(\frac{1}{p}+\frac{1}{q}-1\right)=a^{p}\left(1-1\right)=0. \]

Hence the minimum is \(0\) and the inequality is established. The domain of \(g\) is \(\left\{ a:0<a<\infty\right\}\) and we have \(p>1\), so that for some fixed \(b\), \(g'\left(a\right)=a^{p-1}-b\) is increasing in \(a\). Thus, the minimum we found is unique, so that equality holds only if \(a^{p-1}=b\), which is equivalent to

\[ a^{p-1}=b\implies a^{p/q}=b\implies\left(a^{p/q}\right)^{q}=b^{q}\implies a^{p}=b^{q}. \]

Theorem 3.16 (Hölder’s Inequality) Let \(X\) and \(Y\) be any two random variables, and let \(p\) and \(q\) satisfy Lemma 3.2. Then

\[ \left|\E\left[XY\right]\right|\leq \E\left[\left|XY\right|\right]\leq \left(\E\left[\left|X\right|^{p}\right]\right)^{1/p}\left(\E\left[\left|Y\right|^{q}\right]\right)^{1/q}. \]

(This is Theorem 4.7.2 from Casella and Berger (2002); the following proof is given there.)

Proof. The first inequality follows from \(-\left|XY\right|\leq XY\leq\left|XY\right|\) and Theorem 3.7. To prove the second inequality, define

\[ a=\frac{\left|X\right|}{\left(\E\left[\left|X\right|^{p}\right]\right)^{1/p}}\quad\text{and}\quad b=\frac{\left|Y\right|}{\left(\E\left[\left|Y\right|^{q}\right]\right)^{1/q}}. \]

Applying Lemma 3.2, we get

\[\begin{align*} \frac{1}{p}a^{p}+\frac{1}{q}b^{q} & \geq ab \\ \implies\frac{1}{p}\left(\frac{\left|X\right|}{\left(\E\left[\left|X\right|^{p}\right]\right)^{1/p}}\right)^{p}+\frac{1}{q}\left(\frac{\left|Y\right|}{\left(\E\left[\left|Y\right|^{q}\right]\right)^{1/q}}\right)^{q} & \geq\frac{\left|X\right|\left|Y\right|}{\left(\E\left[\left|X\right|^{p}\right]\right)^{1/p}\left(\E\left[\left|Y\right|^{q}\right]\right)^{1/q}} \\ \implies\frac{1}{p}\frac{\left|X\right|^{p}}{\E\left[\left|X\right|^{p}\right]}+\frac{1}{q}\frac{\left|Y\right|^{q}}{\E\left[\left|Y\right|^{q}\right]} & \geq\frac{\left|XY\right|}{\left(\E\left[\left|X\right|^{p}\right]\right)^{1/p}\left(\E\left[\left|Y\right|^{q}\right]\right)^{1/q}}. \end{align*}\]

Taking the expectation of both sides gives

\[\begin{align*} \E\left[\frac{1}{p}\frac{\left|X\right|^{p}}{\E\left[\left|X\right|^{p}\right]}+\frac{1}{q}\frac{\left|Y\right|^{q}}{\E\left[\left|Y\right|^{q}\right]}\right] & \geq\E\left[\frac{\left|XY\right|}{\left(\E\left[\left|X\right|^{p}\right]\right)^{1/p}\left(\E\left[\left|Y\right|^{q}\right]\right)^{1/q}}\right] \\ \implies\frac{1}{p\E\left[\left|X\right|^{p}\right]}\E\left[\left|X\right|^{p}\right]+\frac{1}{q\E\left[\left|Y\right|^{q}\right]}\E\left[\left|Y\right|^{q}\right] & \geq\E\left[\frac{\left|XY\right|}{\left(\E\left[\left|X\right|^{p}\right]\right)^{1/p}\left(\E\left[\left|Y\right|^{q}\right]\right)^{1/q}}\right] \\ \implies\frac{1}{p}+\frac{1}{q} & \geq\frac{1}{\left(\E\left[\left|X\right|^{p}\right]\right)^{1/p}\left(\E\left[\left|Y\right|^{q}\right]\right)^{1/q}}\E\left[\left|XY\right|\right] \\ \implies1 & \geq\frac{1}{\left(\E\left[\left|X\right|^{p}\right]\right)^{1/p}\left(\E\left[\left|Y\right|^{q}\right]\right)^{1/q}}\E\left[\left|XY\right|\right] \\ \implies\E\left[\left|XY\right|\right] & \leq\left(\E\left[\left|X\right|^{p}\right]\right)^{1/p}\left(\E\left[\left|Y\right|^{q}\right]\right)^{1/q}. \end{align*}\]

Perhaps the most famous special case of Hölder’s Inequality is that for which \(p=q=2\).

Theorem 3.17 (Cauchy-Schwarz Inequality) For any two random variables \(X\) and \(Y\),

\[ \left|\E\left[XY\right]\right|\leq\E\left[\left|XY\right|\right]\leq\left(\E\left[\left|X\right|^{2}\right]\right)^{1/2}\left(\E\left[\left|Y\right|^{2}\right]\right)^{1/2}. \]

(This is Theorem 4.7.3 from Casella and Berger (2002).)

Definition 3.7 A function \(g\left(x\right)\) is convex if \(g\left(\lambda x+\left(1-\lambda\right)y\right)\leq\lambda g\left(x\right)+\left(1-\lambda\right)g\left(y\right)\), for all \(x\) and \(y\), and \(0<\lambda<1\). The function \(g\left(x\right)\) is concave if \(-g\left(x\right)\) is convex.

Theorem 3.18 (Jensen’s Inequality) For any random variable \(X\), if \(g\left(x\right)\) is a convex function, then

\[ \E\left[g\left(X\right)\right]\geq g\left(\E\left[X\right]\right). \]

Equality holds if and only if, for every line \(a+bx\) that is tangent to \(g\left(x\right)\) at \(x=\E\left[X\right]\), \(P\left(g\left(X\right)=a+bX\right)=1\).

3.4 Properties of a random sample

3.4.1 Sums of random variables from a random sample

Definition 3.8 The sample variance is the statistic defined by

\[ S^{2}=\frac{1}{n-1}\sum_{i=1}^{n}\left(X_{i}-\bar{X}\right)^{2}. \]

The sample standard deviation is the statistic defined by \(S=\sqrt{S^{2}}\).

Theorem 3.19 Let \(x_{1},\ldots,x_{n}\) be any numbers and \(\bar{x}=\left(x_{1}+\cdots+x_{n}\right)/n\). Then

1. \(\min_{a}\sum_{i=1}^{n}\left(x_{i}-a\right)^{2}=\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}\),
2. \(\left(n-1\right)s^{2}=\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}=\sum_{i=1}^{n}x_{i}^{2}-n\bar{x}^{2}\).

(This is Theorem 5.2.4 from Casella and Berger (2002).; the following proof is given there.)

Proof. To prove part (1), add and subtract \(\bar{x}\) to get

\[\begin{align*} \sum_{i=1}^{n}\left(x_{i}-a\right)^{2} & =\sum_{i=1}^{n}\left(x_{i}-\bar{x}+\bar{x}-a\right)^{2} \\ & =\sum_{i=1}^{n}\left[\left(x_{i}-\bar{x}\right)+\left(\bar{x}-a\right)\right]\left[\left(x_{i}-\bar{x}\right)+\left(\bar{x}-a\right)\right] \\ & =\sum_{i=1}^{n}\left[\left(x_{i}-\bar{x}\right)^{2}+\left(x_{i}-\bar{x}\right)\left(\bar{x}-a\right)+\left(\bar{x}-a\right)\left(x_{i}-\bar{x}\right)+\left(\bar{x}-a\right)^{2}\right] \\ & =\sum_{i=1}^{n}\left[\left(x_{i}-\bar{x}\right)^{2}+2\left(x_{i}-\bar{x}\right)\left(\bar{x}-a\right)+\left(\bar{x}-a\right)^{2}\right] \\ & =\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}+\sum_{i=1}^{n}\left[2\left(x_{i}-\bar{x}\right)\left(\bar{x}-a\right)\right]+\sum_{i=1}^{n}\left(\bar{x}-a\right)^{2} \\ & =\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}+2\sum_{i=1}^{n}\left(x_{i}\bar{x}-ax_{i}-\bar{x}^{2}+a\bar{x}\right)+\sum_{i=1}^{n}\left(\bar{x}-a\right)^{2} \\ & =\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}+2\left(\bar{x}\sum_{i=1}^{n}x_{i}-a\sum_{i=1}^{n}x_{i}-\sum_{i=1}^{n}\bar{x}^{2}+\sum_{i=1}^{n}a\bar{x}\right)+\sum_{i=1}^{n}\left(\bar{x}-a\right)^{2} \\ & =\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}+2\left(\bar{x}\left(n\bar{x}\right)-a\left(n\bar{x}\right)-n\bar{x}^{2}+na\bar{x}\right)+\sum_{i=1}^{n}\left(\bar{x}-a\right)^{2} \\ & =\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}+2\left(n\bar{x}^{2}-na\bar{x}-n\bar{x}^{2}+na\bar{x}\right)+\sum_{i=1}^{n}\left(\bar{x}-a\right)^{2} \\ & =\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}+2\left(0\right)+\sum_{i=1}^{n}\left(\bar{x}-a\right)^{2} \\ & =\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}+\sum_{i=1}^{n}\left(\bar{x}-a\right)^{2}. \end{align*}\]

It is now clear that the right-hand side is minimized at \(a=\bar{x}\). To prove part (2), take \(a=0\) in the above, i.e.,

\[\begin{align*} \sum_{i=1}^{n}\left(x_{i}-a\right)^{2} & =\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}+\sum_{i=1}^{n}\left(\bar{x}-a\right)^{2} \\ \implies\sum_{i=1}^{n}\left(x_{i}-0\right)^{2} & =\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}+\sum_{i=1}^{n}\left(\bar{x}-0\right)^{2} \\ \implies\sum_{i=1}^{n}x_{i}^{2} & =\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}+\sum_{i=1}^{n}\bar{x}^{2} \\ \implies\sum_{i=1}^{n}x_{i}^{2}-\sum_{i=1}^{n}\bar{x}^{2} & =\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2} \\ \implies\sum_{i=1}^{n}x_{i}^{2}-n\bar{x}^{2} & =\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}. \end{align*}\]

The sample variance is defined as

\[ s^{2}=\frac{1}{n-1}\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}\implies\left(n-1\right)s^{2}=\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}, \]

so the final equality of part (2) holds.

3.4.2 Sampling from the normal distribution

Theorem 3.20 Let \(X_{1},\ldots,X_{n}\) be a random sample from a \(\mathcal{N}\left(\mu,\sigma^{2}\right)\) distribution, and let \(\bar{X}=\left(1/n\right)\sum_{i=1}^{n}X_{i}\) and \(S^{2}=\left[1/\left(n-1\right)\right]\sum_{i=1}^{n}\left(X_{i}-\bar{X}\right)^{2}\). Then

1. \(\bar{X}\) and \(S^{2}\) are independent random variables,
2. \(\bar{X}\) has a \(\mathcal{N}\left(\mu,\sigma^{2}/n\right)\) distribution,
3. \(\left(n-1\right)S^{2}/\sigma^{2}\) has a chi-squared distribution with \(n-1\) degrees of freedom.

(This is Theorem 5.3.1 from Casella and Berger (2002).; the following proof is given there.)

Proof. PROOF GOES HERE

Definition 3.9 Let \(X_{1},\ldots,X_{n}\) be a random sample from a \(\mathcal{N}\left(\mu,\sigma^{2}\right)\) distribution. The quantity \(\left(\bar{X}-\mu\right)/\left(S/\sqrt{n}\right)\) has Student’s \(t\) distribution with \(n-1\) degrees of freedom. Equivalently, a random variable \(T\) has Student’s \(t\) distribution with \(p\) degrees of freedom, and we write \(T\sim t_{p}\) if it has pdf

\[ f_{T}\left(t\right)=\frac{\Gamma\left(\frac{p+1}{2}\right)}{\Gamma\left(\frac{p}{2}\right)}\frac{1}{\left(p\pi\right)^{1/2}}\frac{1}{\left(1+t^{2}/p\right)^{\left(p+1\right)/2}},\quad-\infty<t<\infty. \]

3.4.3 Order statistics

The order statistics of a random sample \(X_{1},\ldots,X_{n}\) are the sample values placed in ascending order. They are denoted by \(X_{\left(1\right)},\ldots,X_{\left(n\right)}\). The order statistics are random variables that satisfy \(X_{\left(1\right)}\leq\ldots\leq X_{\left(n\right)}\), and in particular, \(X_{\left(1\right)}=\underset{1\leq i\leq n}{\min}X_{i}\) and \(X_{\left(n\right)}=\underset{1\leq i\leq n}{\max}X_{i}\).

Theorem 3.21 Let \(X_{1},\ldots,X_{n}\) be a random sample from a discrete distribution with pmf \(f_{X}\left(x_{i}\right)=p_{i}\), where \(x_{1}<x_{2}<\cdots\) are the possible values of \(X\) in ascending order. Define

\[\begin{align*} P_{0} & =0 \\ P_{1} & =p_{1} \\ P_{2} & =p_{1}+p_{2} \\ & \vdots \\ P_{i} & =p_{1}+p_{2}+\cdots+p_{i} \\ & \vdots \end{align*}\]

Let \(X_{\left(1\right)},\ldots,X_{\left(n\right)}\) denote the order statistics from the sample. Then

\[ P\left(\left\{ X_{\left(j\right)}\leq x_{i}\right\} \right)=\sum_{k=j}^{n}\binom{n}{k}P_{i}^{k}\left(1-P_{i}\right)^{n-k} \]

and

\[ P\left(\left\{ X_{\left(j\right)}=x_{i}\right\} \right)=\sum_{k=j}^{n}\binom{n}{k}\left[P_{i}^{k}\left(1-P_{i}\right)^{n-k}-P_{i-1}^{k}\left(1-P_{i-1}\right)^{n-k}\right]. \]

(This is Theorem 5.4.3 from Casella and Berger (2002).)

Proof. PROOF GOES HERE

Theorem 3.22 Let \(X_{\left(1\right)},\ldots,X_{\left(n\right)}\) denote the order statistics of a random sample, \(X_{1},\ldots,X_{n}\), from a continuous population with cdf \(F_{X}\left(x\right)\) and pdf \(f_{X}\left(x\right)\). Then the pdf of \(X_{\left(j\right)}\) is

\[ f_{X_{\left(j\right)}}\left(x\right)=\frac{n!}{\left(j-1\right)!\left(n-j\right)!}f_{X}\left(x\right)\left[F_{X}\left(x\right)\right]^{j-1}\left[1-F_{X}\left(x\right)\right]^{n-j}. \]

(This is Theorem 5.4.4 from Casella and Berger (2002).)

Proof. PROOF GOES HERE

Theorem 3.23 Let \(X_{\left(1\right)},\ldots,X_{\left(n\right)}\) denote the order statistics of a random sample, \(X_{1},\ldots,X_{n}\), from a continuous population with cdf \(F_{X}\left(x\right)\) and pdf \(f_{X}\left(x\right)\). Then the joint pdf of \(X_{\left(i\right)} and X_{\left(j\right)}\), \(1\leq i<j\leq n\), is

\[\begin{align*} f_{X_{\left(i\right)},X_{\left(j\right)}}\left(u,v\right) & =\frac{n!}{\left(i-1\right)!\left(j-1-i\right)!\left(n-j\right)!}f_{X}\left(u\right)f_{X}\left(v\right) \\ &\quad\times\left[F_{X}\left(u\right)\right]^{i-1}\left[F_{X}\left(v\right)-F_{X}\left(u\right)\right]^{j-1-i}\left[1-F_{X}\left(v\right)\right]^{n-j} \end{align*}\]

for \(-\infty<u<v<\infty\).

(This is Theorem 5.4.6 from Casella and Berger (2002).)

Proof. PROOF GOES HERE

3.4.4 Convergence concepts

Theorem 3.24 (Strong Law of Large Numbers) Let \(X_{1},X_{2},\ldots\) be iid random variables with \(\E\left[X_{i}\right]=\mu\) and \(\Var\left(X_{i}\right)=\sigma^{2}<\infty\), and define \(\bar{X}_{n}=\left(1/n\right)\sum_{i=1}^{n}X_{i}\). Then, for every \(\epsilon>0\),

\[ P\left(\lim_{n\rightarrow\infty}\left|\bar{X}_{n}-\mu\right|<\epsilon\right)=1; \]

that is, \(\bar{X}_{n}\) converges almost surely to \(\mu\).

Proof. PROOF GOES HERE

Theorem 3.25 (Central Limit Theorem) Let \(X_{1},X_{2},\ldots\) be a sequence of iid random variables whose mgfs exist in a neighborhood of \(0\) (that is, \(M_{X_{i}}\left(t\right)\) exists for \(\left|t\right|<h\), for some positive \(h\)). Let \(\E\left[X_{i}\right]=\mu\) and \(\Var\left(X_{i}\right)=\sigma^{2}>0\). (Both \(\mu\) and \(\sigma^{2}\) are finite since the mgf exists.) Define \(\bar{X}_{n}=\left(1/n\right)\sum_{i=1}^{n}X_{i}\). Let \(G_{n}\left(x\right)\) denote the cdf of \(\sqrt{n}\left(\bar{X}-\mu\right)/\sigma\). Then, for any \(x\in\mathbb{R}\),

\[ \lim_{n\rightarrow\infty}G_{n}\left(x\right)=\int_{-\infty}^{x}\frac{1}{\sqrt{2\pi}}\mathrm{e}^{-y^{2}/2}\dif y; \]

that is, \(\sqrt{n}\left(\bar{X}_{n}-\mu\right)/\sigma\) has a limiting standard normal distribution.

(This is Theorem 5.5.14 from Casella and Berger (2002); the following proof is given there.)

Proof. Let \(Z\sim\mathcal{N}\left(0,1\right)\), so that the mgf of \(Z\) given by \(M_{Z}\left(t\right)=\mathrm{e}^{0\cdot t+\left(1\cdot t^{2}\right)/2}=\mathrm{e}^{t^{2}/2}\). We will show that, for \(\left|t\right|<h\), the mgf of \(\sqrt{n}\left(\bar{X}_{n}-\mu\right)/\sigma\) converges to \(\mathrm{e}^{t^{2}/2}\).

Define \(Y_{i}=\left(X_{i}-\mu\right)/\sigma\), and let \(M_{Y}\left(t\right)\) denote the common mgf of the \(Y_{i}\text{'s}\), which exists for \(\left|t\right|<\sigma h\) and is given by Theorem 3.9. We have

\[ \frac{X_{i}-\mu}{\sigma}=Y_{i}\implies X_{i}-\mu=\sigma Y_{i}\implies X_{i}=\sigma Y_{i}+\mu, \]

so that

\[\begin{align*} \frac{\sqrt{n}\left(\bar{X}_{n}-\mu\right)}{\sigma} & =\frac{\sqrt{n}\left(\frac{1}{n}\sum_{i=1}^{n}X_{i}-\mu\right)}{\sigma} \\ & =\frac{\sqrt{n}}{\sigma}\left[\frac{1}{n}\sum_{i=1}^{n}\left(\sigma Y_{i}+\mu\right)-\mu\right] \\ & =\frac{\sqrt{n}}{\sigma}\left[\frac{1}{n}\left(\sigma\sum_{i=1}^{n}Y_{i}+n\mu\right)-\mu\right] \\ & =\frac{\sqrt{n}}{\sigma}\left[\frac{\sigma}{n}\sum_{i=1}^{n}Y_{i}+\mu-\mu\right] \\ & =\frac{\sqrt{n}}{\sigma}\left(\frac{\sigma}{n}\sum_{i=1}^{n}Y_{i}\right) \\ & =\frac{1}{\sqrt{n}}\sum_{i=1}^{n}Y_{i}. \end{align*}\]

Then, from the properties of mgfs (see Theorems 3.9 and 3.15), we have

\[ M_{\sqrt{n}\left(\bar{X}_{n}-\mu\right)/\sigma}\left(t\right)=M_{\sum_{i=1}^{n}Y_{i}/\sqrt{n}}\left(t\right)=M_{\sum_{i=1}^{n}Y_{i}}\left(\frac{t}{\sqrt{n}}\right)=\left(M_{Y}\left(\frac{t}{\sqrt{n}}\right)\right)^{n}. \]

We now expand \(M_{Y}\left(t/\sqrt{n}\right)\) in a Taylor series (power series) around 0. We have

\[ M_{Y}\left(\frac{t}{\sqrt{n}}\right)=\sum_{k=0}^{\infty}M_{Y}^{\left(k\right)}\left(0\right)\frac{\left(t/\sqrt{n}\right)^{k}}{k!}, \]

where \(M_{Y}^{\left(k\right)}\left(0\right)=\left(\dif^{k}/\dif t^{k}\right)M_{Y}\left(t\right)\vert_{t=0}\). Since the mgfs exist for \(\left|t\right|<h\), the power series expansion is valid if \(t<\sqrt{n}\sigma h\).

We have

\[\begin{align*} M_{Y}^{\left(0\right)} & =\E\left[Y^{0}\right]=\E\left[1\right]=1, \\ M_{Y}^{\left(1\right)} & =\E\left[Y^{1}\right]=\E\left[\frac{X-\mu}{\sigma}\right]=\frac{1}{\sigma}\left(\E\left[X\right]-\E\left[\mu\right]\right)=\frac{1}{\sigma}\left(\mu-\mu\right)=0, \end{align*}\]

and, noting that

\[ \Var\left(X\right)=\E\left[X^{2}\right]-\left(\E\left[X\right]\right)^{2}\implies\sigma^{2}=\E\left[X^{2}\right]-\mu^{2}\implies\E\left[X^{2}\right]=\mu^{2}+\sigma^{2}, \]

we have

\[\begin{align*} M_{Y}^{\left(2\right)} & =\E\left[Y^{2}\right] \\ & =\E\left[\left(\frac{X-\mu}{\sigma}\right)^{2}\right] \\ & =\frac{1}{\sigma^{2}}\E\left[X^{2}-2\mu X+\mu^{2}\right] \\ & =\frac{1}{\sigma^{2}}\left(\E\left[X^{2}\right]-2\mu\E\left[X\right]+\E\left[\mu^{2}\right]\right) \\ & =\frac{1}{\sigma^{2}}\left(\mu^{2}+\sigma^{2}-2\mu^{2}+\mu^{2}\right) \\ & =\frac{1}{\sigma^{2}}\left(\sigma^{2}\right) \\ & =1. \end{align*}\]

(By construction, the mean and variance of Y are 0 and 1, respectively.) Then, we have

\[\begin{align*} M_{Y}\left(\frac{t}{\sqrt{n}}\right) & =\sum_{k=0}^{\infty}M_{Y}^{\left(k\right)}\left(0\right)\left(\frac{t/\sqrt{n}}{k!}\right)^{k} \\ & =1\frac{\left(t\sqrt{n}\right)^{0}}{0!}+0\frac{\left(t\sqrt{n}\right)}{1!}+1\frac{\left(t\sqrt{n}\right)^{2}}{2!}+\sum_{k=3}^{\infty}M_{Y}^{\left(k\right)}\left(0\right)\frac{\left(t/\sqrt{n}\right)^{k}}{k!} \\ & =T_{2}\left(\frac{t}{\sqrt{n}}\right)+R_{2}\left(\frac{t}{\sqrt{n}}\right), \end{align*}\]

where

\[ T_{2}\left(\frac{t}{\sqrt{n}}\right)=1+\frac{\left(t/\sqrt{n}\right)^{2}}{2!}\quad\text{and}\quad R_{2}\left(\frac{t}{\sqrt{n}}\right)=\sum_{k=3}^{\infty}M_{Y}^{\left(k\right)}\left(0\right)\frac{\left(t/\sqrt{n}\right)^{k}}{k!}. \]

We have \(n>0\), so for fixed \(t\neq 0\), the quantity \(t/\sqrt{n}\rightarrow 0\) as \(n\rightarrow\infty\). Then, noting that \(M_{Y}^{\left(2\right)}\left(0\right)\) exists, it follows from Theorem 3.26 that

\[ \lim_{t/\sqrt{n}\rightarrow0}\frac{M_{Y}\left(t/\sqrt{n}\right)-T_{2}\left(t/\sqrt{n}\right)}{\left(t/\sqrt{n}-0\right)^{2}}=0\implies\lim_{n\rightarrow\infty}\frac{R_{2}\left(t/\sqrt{n}\right)}{\left(t/\sqrt{n}\right)^{2}}=0. \]

Since \(t\) is fixed, we also have

\[ \lim_{n\rightarrow\infty}\frac{R_{2}\left(t/\sqrt{n}\right)}{\left(1/\sqrt{n}\right)^{2}}=\lim_{n\rightarrow\infty}nR_{2}\left(\frac{t}{\sqrt{n}}\right)=0, \]

and this is also true at \(t=0\) since

\[ R_{2}\left(\frac{0}{\sqrt{n}}\right)=R_{2}\left(0\right)=\sum_{k=3}^{\infty}M_{Y}^{\left(k\right)}\left(0\right)\frac{0^{k}}{k!}=\sum_{k=3}^{\infty}M_{Y}^{\left(k\right)}\left(0\right)\cdot0=0. \]

Thus, for any fixed \(t\), we can write

\[\begin{align*} \lim_{n\rightarrow\infty}\left(M_{Y}\left(\frac{t}{\sqrt{n}}\right)\right)^{n} & =\lim_{n\rightarrow\infty}\left[1+\frac{\left(t/\sqrt{n}\right)^{2}}{2}+R_{2}\left(\frac{t}{\sqrt{n}}\right)\right]^{n} \\ & =\lim_{n\rightarrow\infty}\left[1+\frac{t^{2}}{2n}+R_{2}\left(\frac{t}{\sqrt{n}}\right)\right]^{n} \\ & =\lim_{n\rightarrow\infty}\left[1+\frac{1}{n}\left(\frac{t^{2}}{2}+nR_{2}\left(\frac{t}{\sqrt{n}}\right)\right)\right]^{n}. \end{align*}\]

Setting \(a_{n}=\left(t^{2}/2\right)+nR_{2}\left(t/\sqrt{n}\right)\), we have

\[ \lim_{n\rightarrow\infty}a_{n}=\lim_{n\rightarrow\infty}\left[\frac{t^{2}}{2}+nR_{2}\left(\frac{t}{\sqrt{n}}\right)\right]=\lim_{n\rightarrow\infty}\frac{t^{2}}{2}+\lim_{n\rightarrow\infty}nR_{2}\left(\frac{t}{\sqrt{n}}\right)=\frac{t^{2}}{2}+0=\frac{t^{2}}{2}, \]

i.e., the sequence \(a_{1},a_{2},\ldots\) converges to \(a=t^{2}/2\) as \(n\rightarrow\infty\). It follows from Lemma 3.1 that

\[ \lim_{n\rightarrow\infty}\left(M_{Y}\left(\frac{t}{\sqrt{n}}\right)\right)^{n}=\lim_{n\rightarrow\infty}\left[1+\frac{1}{n}\left(\frac{t^{2}}{2}+nR_{2}\left(\frac{t}{\sqrt{n}}\right)\right)\right]^{n}=\lim_{n\rightarrow\infty}\left[1+\frac{a_{n}}{n}\right]^{n}=\mathrm{e}^{a}=\mathrm{e}^{t^{2}/2}. \]

Since \(\mathrm{e}^{t^{2}/2}\) is the mgf of the \(\mathcal{N}\left(0,1\right)\) distribution, the theorem is proved.

Definition 3.10 If a function \(g\left(x\right)\) has derivatives of order \(r\), that is, \(g^{\left(r\right)}\left(x\right)=\frac{\dif^{r}}{\dif x^{r}}g\left(x\right)\) exists, then for any constant \(a\), the Taylor polynomial of order \(r\) about \(a\) is

\[ T_{r}\left(x\right)=\sum_{i=0}^{r}\frac{g^{\left(i\right)}\left(a\right)}{i!}\left(x-a\right)^{i}. \]

Theorem 3.26 If

\[ g^{\left(r\right)}\left(a\right)=\frac{\dif^{r}}{\dif x^{r}}g\left(x\right)\Bigr\vert_{x=a} \]

exists, then

\[ \lim_{x\rightarrow a}\frac{g\left(x\right)-T_{r}\left(x\right)}{\left(x-a\right)^{r}}=0. \]

(This is Theorem 5.5.21 from Casella and Berger (2002)).

References

Casella, G., and R.L. Berger. 2002. Statistical Inference. Duxbury Advanced Series in Statistics and Decision Sciences. Thomson Learning. https://books.google.com/books?id=0x\_vAAAAMAAJ.