[Home]
Last updated on: Mon Aug 10 12:15:29 IST 2026

Sampling distributions for normal mean and variance

$\newcommand{\v}[1]{\boldsymbol{#1}}$ $\newcommand{\k}[1]{\chi^2_{(#1)}}$ $\newcommand{\cov}{\mathrm{cov}}$ Sampling distributions for normal mean and variance Every random variable has a distribution. If $X_1,...,X_n$ is a random sample, and $T:{\mathbb R}^n\rightarrow{\mathbb R}$ is any given function, then $T(X_1,...,X_n)$ is a random variable. It is called a statistic. Its distribution is called its sampling distribution. We shall learn about the sampling distribution of the sample mean and sample variance when $X_i$ s are iid $N(\mu,\sigma^2).$ In order to derive these we need to first know about the multivariate normal distribution.

1. Multivariate normal

You have already learned about normal and bivarate normal distributions in Probability I. These are all special cases of the multivariate normal distribution.

Definition: Multivariate normal distribution Let $\v X=(X_1,...,X_n)'$ be a random vector with $X_i$'s iid $N(0,1).$ Let $A_{d\times n}$ be any fixed matrix and $\v \mu\in{\mathbb R}^d$ be any fixed vector. Then the distribution of the random vector $A\v X + \v \mu$ is called a $d$-dimensional multivariate normal distribution and is denoted by $N_d(\v \mu, \Sigma),$ where $\Sigma=AA'.$
Notice a subtle point here: Though we defined the random vector using $A,$ yet we labelled it with $AA',$ and not $A$ itself. This is because the distribution of $A\v X + \v \mu$ will depend on $A$ only through $AA'.$ In other words, if I use $A_1$ and you use $A_2$ such that $A_1A_1' = A_2A_2',$ then the $A_1\v X + \v \mu$ and $A_2\v X + \v \mu$ will have the same distribution. We shall prove this point in the final section of this page.

TheoremLet $\v X = \left[\begin{array}{ccccccccccc}X_1\\\vdots\\X_d \end{array}\right]$. Then $\v X\sim N_d(\v 0, I)$ if and only if $X_1,...,.X_n$ are iid $N(0,1)$.

Proof:Only if part: Take $A = I_d$ and $\v \mu = \v 0_{d\times 1}.$ Then by the above definition $$A\v X + \v \mu = \v X\sim N_d(\v 0, I),$$ because $AA' = I.$

If part: Shall do later in the final section of this page. [QED]

TheoremIf $\v X\sim N_d(\v \mu, \Sigma),$ then $E(\v X) = \v \mu$ and $V(\v X) = \Sigma.$

Proof:Here $\v X = \v \mu + A\v Z$ for some matrix $A$ where $\Sigma = AA'$ and the components of $\v Z$ are iid $N(0,1).$

Hence $E(\v Z) = \v 0$ and $V(\v Z) = I.$

So $E(\v X) = \v \mu + A E(\v Z) = \v \mu$ and $V(\v X) = A V(\v Z)A' = AA' = \Sigma.$ [QED]

TheoremIf $\v X\sim N_d(\v \mu, \Sigma),$ and $B_{r\times d}$ is any fixed matrix and $\v c\in{\mathbb R}^r$ is any fixed vector, then $BX+\v c\sim N_r(B\v\mu+\v c, B \Sigma B').)$

Proof:Here $\v X = \v \mu + A\v Z$ for some matrix $A$ where $\Sigma = AA'$ and the components of $\v Z$ are iid $N(0,1).$

So $B\v X + \v c = B(\v \mu + A\v Z) +\v c= B\v \mu+\v c + BA\v Z\sim N_r(B\v \mu+\v c,BAA'B')\equiv N_r(B\v \mu+\v c,B \Sigma B'),$ as required. [QED]

We can easily compute the marginals of a multivariate normal distribution using this theorem.

EXAMPLE 1:  Let $\v X = \left[\begin{array}{ccccccccccc}X\\Y\\Z \end{array}\right]\sim N_3\left(\left[\begin{array}{ccccccccccc}1\\2\\3 \end{array}\right],\left[\begin{array}{ccccccccccc}2 & 0 & 1\\0 & 20 & 2\\1 & 2 & 100 \end{array}\right]\right).$ Find the distributions of $Y$ and $(X,Z).$

SOLUTION: Take $B = \left[\begin{array}{ccccccccccc}0 & 1 & 0 \end{array}\right].$ Then $B\v X = Y\sim N(2,20).$

Again, taking $B = \left[\begin{array}{ccccccccccc}1 & 0 & 0\\0 & 0 & 1 \end{array}\right]$, we have $B\v X = \left[\begin{array}{ccccccccccc}X\\Z \end{array}\right]\sim N_2\left(\left[\begin{array}{ccccccccccc}1\\3 \end{array}\right],\left[\begin{array}{ccccccccccc}2 & 1\\1 & 100 \end{array}\right]\right)$. ■

Thus, we see that the various marginals are also normal with the "corresponding" parts of the original mean vector and covariance matrix.

TheoremIf $\v X\sim N_d(\v \mu, \Sigma),$ then $X_1,...,.X_d$ are (mutually) independent iff $\Sigma$ is diagonal.

Proof:Will be done in the final section of this page.[QED]

1.1. Problem set

Let $\left[\begin{array}{ccccccccccc}W\\X\\Y\\Z \end{array}\right]\sim N_3\left(\left[\begin{array}{ccccccccccc}1\\-2\\0\\3 \end{array}\right],\left[\begin{array}{ccccccccccc}30 & 0 & 1 & 2\\0 & 20 & 5 & 1\\1 & 5 & 100 & 1\\2 & 1 & 1 & 3 \end{array}\right]\right).$

EXERCISE 1: Find the distribution of $Y.$

EXERCISE 2: Find the distribution of $(X,Z).$

EXERCISE 3: Find the distribution of $(X+Y,X+W).$

EXERCISE 4: If $X_1,X_2,X_3$ are iid $N(0,1),$ then show that $X_1+X_2+X_3$ and $X_1-2X_2+X_3$ are independent.

EXERCISE 5: If $\v X\sim N_d(\v \mu, I)$ and $\v a, \v b\in{\mathbb R}^d$ are two fixed vectors with $\v a'\v b = 0$ (i.e., they are orthogonal to each other), then show that $\v a'\v X$ and $\v b'\v X$ are independent.

EXERCISE 6: Show that $N_d(\v 0, I)$ has joint density $f(\v x) = (2\pi)^{-\frac d2} e^{-\frac 12\v x'\v x} $ for $\v x\in{\mathbb R}^d.$

Hint:

The components are iid $N(0,1)$. Multiply their densities.

EXERCISE 7: Show that if $\Sigma$ is pd, then $N_d(\v \mu, \Sigma)$ has density $$f(\v x) = (2\pi det(\Sigma))^{-\frac 12} e^{-\frac 12(\v x-\v \mu)' \Sigma ^{-1}(\v x-\v \mu) } \mbox{ for }\v x\in{\mathbb R}^d.$$

[Hint]

Since $\Sigma$ is pd, $\Sigma = AA'$ for a nonsingular matrix $A.$

So you can think that $X = \v \mu + A\v Z$ where $\v Z\sim N_d(\v 0, I).$

The last exercise gives density of $\v Z.$

The transform $X = \v \mu + A\v Z$ is a bijection. Apply Jacobian formula.

2. Sampling distributions for normal sample

Let $X_1,...,X_n$ be a random sample from $N(\mu, \sigma^2).$ Then we are interested in finding the distributions of $\bar X$ and $S^2 = \sum (X_i-\bar X)^2.$

We shall present the results as three theorems.

2.1. Sampling distribution of $\bar X$

Theorem 1 Let $X_1,...,X_n$ be a random sample from $N(\mu, \sigma^2).$ Then $\bar X\sim N\left( \mu, \frac{\sigma^2}{n}\right).$

Proof: Note that $\bar X$ is a linear combination of $X_1,...,X_n.$ So $\bar X$ must be a normal random variable. Its parameters will be $$E(\bar X) = E\left(\frac 1n\sum_1^n X_i\right) = \frac 1n(\underbrace{\mu+\cdots+\mu}_{n\mbox{ times}}) = \mu,$$ and $$V(\bar X) = V\left(\frac 1n\sum_1^n X_i\right) = \frac{1}{n^2}(\underbrace{\sigma^2+\cdots+\sigma^2}_{n\mbox{ times}}) = \frac{\sigma^2}{n},$$ since the $X_i$'s are independent (making the covariance terms zero).

This completes the proof. [QED]

2.2. Sampling distribution of $S^2$

If you feel exhausted, just skim through this part!
Next we shall find the distribution of $S^2 = \sum (X_i-\bar X)^2.$ We need the following definition for this.

Definition: $\chi^2$-distributionIf $U_1,...,U_k$ are iid $N(0,1),$ then the distribution of $\sum_{i=1}^k U_i^2$ is called (central) chi-square with $k$ degrees of freedom. This distribution is denoted by $\k k$.
Thus, we say $X\sim\k k,$ then you can assume that there are $k$ random variables $U_1,...,U_k\sim N(0,1)$ such that $X = U_1^2+\cdots+U_k^2.$

It is good to know how the densities of the $\k k$-distribution look like for different degrees of freedom:
Some $\k k$ densities

Theorem 2 Let $X_1,...,X_n$ be a random sample (i.e., iid) from $N(\mu, \sigma^2).$ Let $S^2=\sum_1^n (X_i-\bar X)^2.$ Then $\frac{S^2}{\sigma^2}\sim \k {n-1}.$
The proof is somewhat long. So let us first understand the main line of thought running behind the proof. We want to prove that $\frac{S^2}{\sigma^2}$ has $\k{n-1}$ distribution. Now $\k{n-1}$ is the distribution of the sum of $n-1$ squares. But the way we have defined it $S^2$ is the sum of $n$ squares. So we need a way to rewrite $S^2$ as a sum of $n-1$ squares. This is just algebraic manipulation and is provided by the following result.

Lemma Let $x_1,x_2,...$ be any sequence of numbers. Let $\bar x_k=\frac 1k\sum _1^kx_i$ and $S_k^2=\sum_{i=1}^k (x_i-\bar x_k)^2$ for $k=1,2,...$.

Then for $k=1,2,...$ $$S_k^2 = \sum_{i=1}^{k-1} \frac{i}{i+1}(x_{i+1}-\bar x_i)^2.$$

Proof:(Induction on $k$):

Basis: For $k=1$, we have $S_1^2 = \sum_{i=1}^1 (x_1-\bar x_1)^2 = 0,$ since $\bar x_1 = x_1.$

Also the rhs is a sum from $i=1$ to $0 (< 1),$ and hence is 0.

Hypothesis: We assume the result for $k=1,2,...,n$ for some $n\geq 1.$

Step: Shall show the result for $k=n+1.$

Now $$\begin{eqnarray*} S_{n+1}^2 & = & \sum_{i=1}^{n+1} (X_i-\bar X_{n+1})^2\\ & = & \sum_{i=1}^{n+1} \left(X_i-\frac{n\bar X_n+X_{n+1}}{n+1}\right)^2\\ & = & \sum_{i=1}^{n+1} \left(\underbrace{X_i-\bar X_n}_a - \underbrace{\frac{X_{n+1}-\bar X_n}{n+1}}_b\right)^2\mbox{(Be careful!)}\\ & & \mbox{Applying $(a-b)^2$ formula,}\\ & = & \sum_{i=1}^{n+1} \left[ (X_i-\bar X_n)^2 + \frac{(X_{n+1}-\bar X_n)^2}{(n+1)^2} -2 \frac{(X_i-\bar X_n)(X_{n+1}-\bar X_n)}{n+1} \right]\\ & = & \sum_{i=1}^{n+1} (X_i-\bar X_n)^2 + \frac{(X_{n+1}-\bar X_n)^2}{n+1} -\frac{2(X_{n+1}-\bar X_n)}{n+1} \sum_{i=1}^{n+1}(X_i-\bar X_n) \\ & = & \big[S_n^2+ (X_{n+1}-\bar X_n)^2\big] + \frac{(X_{n+1}-\bar X_n)^2}{n+1} -\frac{2(X_{n+1}-\bar X_n)}{n+1} (X_{n+1}-\bar X_n)~~\left[\mbox{$\because\sum_1^n (X_i-\bar X_n)=0.$}\right] \\ & = & S_n^2+ (X_{n+1}-\bar X_n)^2 - \frac{(X_{n+1}-\bar X_n)^2}{n+1} \\ & = & S_n^2+ \frac{n}{n+1}(X_{n+1}-\bar X_n)^2. \end{eqnarray*}$$ Now, by the induction hypothesis, we have $$S_n^2 =\sum_{i=1}^{n-1} \frac{i}{i+1}(x_i-\bar x_n)^2.$$ So we have $$S_{n+1}^2 =\sum_{i=1}^{n-1} \frac{i}{i+1}(x_i-\bar x_n)^2+\frac{n}{n+1}(X_{n+1}-\bar X_n)^2 = \sum_{i=1}^n \frac{i}{i+1}(x_i-\bar x_n)^2,$$ as required.

So the result is proved by the principle of mathematical induction. [QED]

Now we are in a position to prove Theorem 2.

Proof of Theorem 2:

The theorem expresses $S_n^2$ as $\sum_{i=1}^{n-1} U_i^2,$ where $U_i = \sqrt{\frac{i}{i+1}} (X_{i+1}-\bar X_i).$

Hence the required $\k{n-1}$ distribution will be proved once we show that $U_1,...,U_{n-1}$ are iid $N(0,1).$

Notice that the $U_i$'s are all linear combinations of $X_1,...,X_n.$ So $(U_1,...,U_{n-1})'$ has a multivariate normal distribution.

So enough to show that $E(U_i) = 0$, $V(U_i) = 1$ and for $i\neq j$ $\cov(U_i,U_j) = 0.$ This completes the proof of Theorem 2.

[QED]

2.3. Independence of $\bar X $ and $S^2$

If you feel exhausted, just skim through this part!
Theorem 3 Let $X_1,...,X_n$ be a random sample (i.e., iid) from $N(\mu, \sigma^2).$ Let $\bar X=\frac 1n\sum _1^nX_i$ and $S^2=\sum_1^n (X_i-\bar X)^2.$ Then $\bar X$ and $S^2$ are independent.

Proof: In the last proof we showed that $S^2$ is a function of $U_1^2,...,U_{n-1}^2.$

So it is enough to show that $\bar X$ is independent of $(U_1,...,U_{n-1}).$

Since $\bar X$ as well as the $U_i$'s are linear combinations of $X_1,...,X_n,$ hence $(\bar X,U_1,...,U_{n-1})$ has multivariate normal distribution. So to prove independence, it is enough to show that $\cov(\bar X, U_i) = 0$ for $i=1,...,n-1.$

Now $$\begin{eqnarray*} \cov(\bar X, U_i) & = & \sqrt{\frac{i}{i+1}}\cov(\bar X, X_{i+1}-\bar X_i)\\ & = & \sqrt{\frac{i}{i+1}}\big[\cov(\bar X, X_{i+1})-\cov(\bar X,\bar X_i) \big]\\ & = & \sqrt{\frac{i}{i+1}}\left[ \frac 1n\cov(X_1+\cdots X_n, X_{i+1})-\frac{1}{ni}\cov(X_1+\cdots+X_n,X_1+\cdots+\cdots X_i) \right]\\ & = & \sqrt{\frac{i}{i+1}}\left[ \frac{\sigma^2}{n}-\frac{\sigma^2}{n} \right] = 0, \end{eqnarray*}$$ as required. [QED]

2.4. Problem set

EXERCISE 8: If $X_1,...,X_{100}$ are iid $N(3,4^2),$ then what is the sampling distribution of sample mean?

EXERCISE 9:  If $(X_1,...,X_{100})$ is a random sample from $N(3,2^2),$ then what is the probability that the sample variance exceeds 5? Express your answer in terms of $F_k(x)$, the distribution function of $\k k$ distribution. Assume that the sample variance is computed with $n-1$ in the denominator.

EXERCISE 10: If $X_1,...,X_n$ are iid $N(2,3^2).$ Find $P(\bar X \in (1,3))$ in terms of the standard normal distribution function $\Phi(\cdot).$

EXERCISE 11: If $X_1,X_2,...$ are iid $N(0,1)$ and $\bar X_n = \frac 1n\sum_1^n X_i$ and $S_n^2 = \sum_1^n (X_i-\bar X_n)^2,$ then which of the following statements is/are true?

You will need the following definiton to solve the next problem.

Definition: Noncentral $\chi^2$-distributionIf $U_1,...,U_k$ are independent $N(\mu_i,1),$ then the distribution of $\sum_{i=1}^k U_i^2$ is called noncentral chi-square with $k$ degrees of freedom and noncentrality $\sum_{i=1}^k \mu_i^2.$. This distribution is sometimes denoted by $\k k\left(\sum_i \mu_i^2\right)$.

EXERCISE 12:  $X_i$ s iid $N(0,1).$ What will the distribution of $\sum_1^n (X_i-a)^2$ be, where $a\in{\mathbb R}$ is a fixed number?

3. $t$ distribution

Definition: $t$-distribution If $X\sim N(0,1)$ and $Y\sim\k n$ and they are independent, then the distribution of $X/(\sqrt{Y/n})$ is called $t$-distribution with $n$ degrees of freedom.

It should not be difficult to see that $t$-density is symmetric around 0. Using Jacobian formula (and some patience) once may show that the density of $t_{(n)}$-distribution is of the form $f_n(t)=\mbox{const } (t^2+n)^{-\frac{n+1}{2}}.$ The densities are much like the $N(0,1)$ density. They lie somewhere between the Cauchy density and $N(0,1)$ density.
Some $t$-densities
As the degrees of freedom increase to $\infty,$ the $t$-distribution approaches $N(0,1).$ For degrees of freedom more than 40, the $t$-density is virtually indisguishable from the $N(0,1)$ density.

3.1. Problem set

EXERCISE 13: Consider the density of $t_{(k)}$-distribution. Do you recognise the $t_{(1)}$-distribution as something already familiar?

EXERCISE 14: Let $X_1,...,X_n$ be a random sample from $N(\mu,\sigma^2).$ Then what is the distribution of $$\frac{\sqrt n(\bar X-\mu)}{\sqrt{\sum(X_i-\bar X)^2/(n-1)}}?$$

EXERCISE 15: Let $X_1,...,X_{100}$ be a random sample from $N(\mu,\sigma^2).$ Find $a>0$ such that $P(\mu\in (\hat \mu-a \hat \sigma^2,\hat \mu+a \hat \sigma^2)) = 0.95,$ where $\hat \mu$ is the sample mean and $\hat \sigma^2$ is the sample variance (with $n-1$ in the denominator). Express your answer in terms of the distribution function, $F_k(\cdot)$ of the $t_{(k)}$-distribution.

4. $F$ distribution

Definition: $F$-distribution If $X\sim \k m$ and $Y\sim \k n$ are independent random variables, then the distribution of $\frac{X/m}{Y/n}$ is called $F$-distribution with numerator degrees of freedom $m$ and denominator degrees of freedom $n.$
The density of the $F_{m,n}$-distribution is of the form $$f(x) =\left\{\begin{array}{ll}\mbox{const }x^{\frac m2-1}(mx+n)^{-\frac{m+n}{2}}&\text{if }x>0\\ 0&\text{otherwise.}\end{array}\right..$$
Some $F$-densities

4.1. Problem set

EXERCISE 16: Let $X_1,...,X_m$ and $Y_1,...,Y_n$ be random samples from $N(\mu_1,\sigma^2)$ and $N(\mu_2,\sigma^2)$, respectively (same $\sigma^2$). Then what is the distribution of $$\frac{\sum(X_i-\bar X)^2/(m-1)}{\sum(Y_i-\bar Y)^2/(n-1)}?$$

EXERCISE 17: If $X\sim t_{(1)},$ then what is the distribution of $X^2?$

EXERCISE 18: If $X\sim F_{(2,3)},$ then what is the distribution of $\frac 1X?$

5. Multivariate mgf

We had left a number of results unproved in the first section of this page. To prove them, we need a mathematically convenient way to express $N_d(\v \mu, \Sigma)$ distribution. Unfortunately, this distribution may not always have a density, and its distribution function is not easy to write down (even for univariate normal the distribution function cannot be written in a closed form). So in order to work with the multivariate normal distribution, we shall use its Moment generating function (mgf).

We quickly recall the definition of mgf that you have learned in Probability I.

Definition: (Multivariate) mgf The (multivariate) mgf of a random vector $\v X$ is $$M(\v t) = E(e^{\v t'\v X}).$$ If we write $\v X = (X_1,...,X_d)'$ and $\v t = (t_1,...,t_d)',$ then we have $$M(t_1,...,t_d) = E(e^{t_1X_1+\cdots+t_dX_d}).$$
It enjoys all the important properties of univariate mgf.
Determines the distributionIf $\v X, \v Y$ have mgf's matching over a neighbourhood of $\v 0,$ then they must have the same distribution.

Proof:Not to be done in this course.[QED]

Sum of IndependentsIf $\v X, \v Y$ are independent random vectors of same dimension with mgf's $M_X(\v t)$ and $M_Y(\v t),$ then $\v X+\v Y$ has mgf $M_X(\v t)M_Y(\v t).$

Proof: $$\begin{eqnarray*} M_{\v X+\v Y}(\v t) & = & E(e^{\v t'(\v X + \v Y)})\\ & = & E(e^{\v t'\v X}e^{\v t'\v Y)})\\ & = & E(e^{\v t'\v X})E(e^{\v t'\v Y})~~\left[\mbox{$\v X, \v Y$ independent}\right]\\ & = & M_X(\v t)M_Y(\v t). \end{eqnarray*}$$ [QED]

List of Independents If $\v X, \v Y$ are independent random vectors (of possibly different dimensions) with mgf's $M_X(\v s)$ and $M_Y(\v t),$ then $\v Z=\left[\begin{array}{ccccccccccc}\v X\\\v Y \end{array}\right]$ has mgf $M_Z(\v s, \v t) = M_X(\v s)M_Y(\v t).$

Proof: $$\begin{eqnarray*} M_{\v Z} (\v s, \v t) & = & E\left[\exp\left(\left[\begin{array}{ccccccccccc}\v s' & \v t' \end{array}\right]\left[\begin{array}{ccccccccccc}\v X\\\v Y \end{array}\right] \right) \right]\\ & = & E(e^{\v s'\v X}e^{\v t'\v Y)})\\ & = & E(e^{\v s'\v X})E(e^{\v t'\v Y})~~\left[\mbox{$\v X, \v Y$ independent}\right]\\ & = & M_X(\v s)M_Y(\v t). \end{eqnarray*}$$ [QED]

5.1. Problem set

EXERCISE 19: If $X_1,X_2,X_3$ are iid $N(0,1),$ then find the mgf $M(\v t)$ of $\v X = \left[\begin{array}{ccccccccccc}X_1\\X_2\\X_3 \end{array}\right].$

Hint:

$N(0,1)$ has mgf $e^{t^2/2}$ for $t\in{\mathbb R}.$

EXERCISE 20: Find the mgf of $N_d(\v 0, I)$ distribution.

EXERCISE 21: If $\v X_{d\times 1}$ has mgf $M(\v t)$, then find the mgf of $A\v X+\v b$ where $A_{n\times d}$ and $\v b_{n\times 1}$ are fixed.

6. The proofs

In this section we shall finally prove the leftovers from the first section:
  1. If $X_1,...,X_n$ are iid $N(0,1)$, and $\v X = \left[\begin{array}{ccccccccccc}X_1\\\vdots\\X_n \end{array}\right],$ then for any fixed $A_{d\times n}$ and $\v \mu_{d\times 1}$ the distribution of $A\v X + \v \mu$ depends only on $AA'$ and $\v \mu$ (and not on $A$ directly).
  2. If $\v X\sim N_d(\v \mu, \Sigma),$ then $X_1,...,.X_d$ are (mutually) independent iff $\Sigma$ is diagonal.
  3. If $\v X\sim N_d(\v 0, I),$ then the components of $\v X$ are iid $N(0,1).$
The last fact is actually a special case of the second.

6.1. Proof 1

The first fact is a direct consequence of the following theorem.
Theorem Let $\v X$ be a $n$-dimensional random vector with components iid $N(0,1).$ Let $A_{d\times n}$ be any fixed matrix and $\v b\in{\mathbb R}^d$ be any fixed vector. Then the MGF of the random vector $A\v X + \v b$ is given by $$M(\v t) = e^{\v b'\v t+\frac 12\v t' AA'\v t}\mbox{ for }\v t\in{\mathbb R}^d.$$

Proof: Recall from Probability I that mgf of each $X_i$ is $E(e^{sX_1})=e^{s^2/2}$ for $s\in{\mathbb R}.$

So the mgf of $\v X = (X_1,...,X_n)'$ is $$\begin{eqnarray*} M_{\v X}(\underbrace{s_1,...,s_n}_{\v s})=E(e^{s_1X_1+\cdots+s_nX_n}) & = & E(e^{s_1X_1})\cdots E(e^{s_nX_n}) ~~\left[\mbox{$\because$independent}\right] \\ & = & e^{\frac 12(s_1^2+\cdots+s_n^2)} = e^{\v s'\v s/2} \end{eqnarray*}$$ for all $\v s=(s_1,...,s_n)'\in{\mathbb R}^n.$

So mgf of $\v Y = A\v X + \v b$ is $$\begin{eqnarray*} M_{\v Y}(\v t) & = & E(e^{\v t'(A\v X+\v b)})\\ & = & E(e^{\v t'A\v X+\v t'\v b})\\ & = & e^{\v t'\v b}E(e^{\v t'A\v X})\\ & = & e^{\v t'\v b}E(e^{(A'\v t)'\v X})\\ & = & e^{\v t'\v b}e^{(A'\v t)'(A'\v t)/2}\\ & = & e^{\v t'\v b+\v t' AA'\v t/2}, \end{eqnarray*}$$ as required. [QED]

Since the MGF is defined over a neighbourhood of the origin (in fact everywhere), the MGF characterises the distribution. Thus, the distribution is characterised by $\v b$ and $AA'.$

Thus, we have also proved:

Theorem $N_d(\v \mu, \Sigma)$ has mgf $M(\v t) = e^{\v t'\v \mu + \frac 12 \v t' \Sigma \v t}$ for $\v t\in{\mathbb R}^d.$

6.2. Proof 2

A little fact from linear algebra:
Fact from linear algebra If $A,B$ are two $n\times n$ symmetric matrices for which $\forall \v x\in{\mathbb R}^d~~\v x'A\v x = \v x' B\v x,$ then $A=B.$

Proof:Let $\{\v e_1,...,\v e_n\}$ be the canonical basis. Then for $i\in\{1,...,n\}$ we have $\v e_i'A\v e_i = \v e_i'B\v e_i$, or $a_{ii} = b_{ii}.$

Similarly, $\forall i,j\in\{1,...,n\}$ if we take $\v x = \v e_i+\v e_j,$ then $\v x'A\v x = \v x' B\v x$, or $$a_{ii}+a_{ij}+a_{ji}+a_{jj}=b_{ii}+b_{ij}+b_{ji}+b_{jj}.$$ Since the diagonal entries match, have $a_{ij}+a_{ji}=b_{ij}+b_{ji}.$

Since $A,B$ are symmetric, hence this forces $a_{ij} = b_{ij}.$ [QED]

TheoremIf $\v X\sim N_d(\v \mu, \Sigma),$ then $X_1,...,.X_n$ are (mutually) independent iff $\Sigma$ is diagonal.

Proof:We already know that $X_i\sim N(\mu_i, \sigma_{ii}),$ where $\mu_i$ is the $i$-th entry of $\v \mu,$ and $\sigma_{ii}$ is the $i$-th diagonal entry of $\Sigma.$

So mgf of $X_i$ is $M(t) = e^{t \mu_i + \sigma_{ii} t^2/2}$ for $t\in{\mathbb R}.$

If the $X_i$ s were independent, then the mgf of $\v X$ would have been $M(t_1)\cdots M(t_d) = e^{\v t' \v\mu +\frac 12\v t' diag(\Sigma)\v t}$ for $\v t\in{\mathbb R}^d,$ where $diag(\Sigma)$ is the diagonal part of $\Sigma.$

Also mgf of $\v X$ is $e^{\v t'\v \mu + \frac 12\v t'\Sigma\v t}$ for $\v t\in{\mathbb R}^d.$

Thus, $X_i$ s are independent iff $$e^{\v t'\v \mu + \frac 12\v t'\Sigma\v t} = e^{\v t' \v\mu +\frac 12\v t' diag(\Sigma)\v t},$$ and this must hold for all $\v t\in{\mathbb R}^d.$

So we must have $\v t'\Sigma\v t=\v t'diag(\Sigma)\v t.$ Since $\Sigma$ is symmetric, hence the fact from linear algebra forces $\Sigma=diag(\Sigma),$ i.e., $\Sigma$ must be a diagonal matrix, as required. [QED]

6.3. Problem set

EXERCISE 22: $(X,Y)$ has mgf $$M(s,t) = \exp(s-t+20s^2+42t^2-12st)\mbox{ for }(s,t)\in{\mathbb R}^2.$$ Find the distribution of $(X,Y).$

EXERCISE 23: Write the density of $X$ if it has mgf $M(t) = \exp(2t+3t^2)$ for $t\in{\mathbb R}.$