[Home]

Independence

Independence

Independence

Two events

Definition: Independence Suppose we have a random experiment with sample space $\Omega,$ and two events $A,B\subseteq\Omega.$ We shall say that $A,B$ are independent if $P(A\cap B) = P(A)P(B).$
It is easy to see why this definition is natural. For example, consider the statement
"Lefthandedness is indepdendent of a person's gender."
If there are about 5% lefties in the entire population, then this statement will imply that these 5% is more or less equally divided among men and women. In particular, if there are 40% men and 60% women, then the proportion of female lefties should be 5% of 60%, i.e., 5% $\times$ 60%. Thus, the multiplication comes naturally.

Here is a more mathematical example.

EXAMPLE 1:  Suppose that we toss a fair coin twice. So the sample space is $\Omega = \{HH, HT, TH, T T\}.$ Let $A$ be the event that the first toss shows $H,$ and $B$ be the event that the second toss shows $H.$ In other words, $A=\{HH,HT\}$ and $B=\{HH,TH\}.$ As the outcome of one toss can have no influence on the outcome of another, so intuitively $A,B$ should be "independent". From the definition also $P(A\cap B) = P(\{HH\}) = \frac 14 = \frac 12\times \frac 12 = P(A)P(B).$ ■

EXERCISE 1: (Easy) Consider a single roll of a fair die. Write down two events, $A,B$ such that $A,B$ are independent. Also, write down two events $C,D$ that are dependent.

Sometimes two events that are somehow related actually turns out to be indepedent. Here is an example.

EXAMPLE 2:  A fair coin is tossed twice. The outcome of the first toss is written down. If the second toss shows a head, then the first outcome is written again. If the second outcome is a tail, then the opposite of the first outome is written. For example, if the both the outcomes are tails, then we first write tail (the first outcome), and then we write head (opposite of the first outcome). Let $A$ be the event that we first write "head", and $B$ be the event that we write "head" in the second position. Are $A,B$ independent?

SOLUTION: Apparently the first toss plays a role in both the writings. But still a simple computation shows that $P(A)=P(B)=\frac 12$ and $P(A\cap B)=\frac 14.$ So we have independence. ■

EXERCISE 2: (Easy) Same problem as above, but this time the coin is biased. Say, $P(H)=0.6.$

[Hint]

Possible outcomes are still $HH, HT, TH$ and $TT.$ Let's work out the probabilities.

$P(HH) = P($ first toss gives head, second toss is head$) = 0.6^2 = 0.36.$

$P(HT) = P($ first toss gives head, second toss gives tail$) = 0.6\times(1-0.6) = 0.24.$

$P(TH) = P($ first toss gives tail, second toss gives tail$) = (1-0.6)^2 = 0.16.$

$P(TT) = P($ first toss gives tail, second toss gives head$) = (1-0.6)\times 0.6 = 0.24.$

So $P(A) = P(\{HH, HT\}) = 0.36+0.24 = 0.6.$

$P(B) = P(\{HH, TH\}) = 0.36+0.16 = 0.52.$

Now $P(A\cap B) = P(HH) = 0.36 \neq P(A)P(B).$

So $A,B$ are not independent.

More than two events

When you have more than two events, the definition of independence becomes a bit counter-intuitive.

Definition: Mutual independence Suppose we have a random experiment with sample space $\Omega,$ and let $A_1,...,A_n\subseteq\Omega$ be any $n$ events. We shall say that these events are mutually independent if $$ \forall \mbox{ nonempty } I\subseteq\{1,...,n\}~~P(\cap_{i\in I} A_i) = \prod_{i\in I} P(A_i). $$
This may look scary. But it only means that if you take any number of the $A_i$'s, say $A_1, A_4$ and $A_7,$ then their probabilities multiply: $P(A_1\cap A_4\cap A_7) = P(A_1)P(A_4)P(A_7).$ Notice that it is not enough to have each pair $A_i,A_j$ to be independent according to the first definition. Indeed, this case has a special name, pairwise independence:
Definition: Pairwise independence Suppose we have a random experiment with sample space $\Omega,$ and let $A_1,...,A_n\subseteq\Omega$ be any $n$ events. We shall say that these events are pairwise independent if $$ \forall i\neq j\in\{1,...,n\}~~P(A_i\cap A_j) = P(A_i)P(A_j). $$

"Mutual independence" is what we intuitively feel when we think that the occurencess of $A_1,...,A_n$ do not influence each other. The following interesting example shows why pairwise independence is not enough.

EXAMPLE 3:  Consider the random experiment where a fair coin is tossed thrice. For $i\neq j\in\{1,2,3\},$ let $A_{ij}$ be the event that the $i$-th and $j$-th tosses have the same outcome. Do you intuitively feel that $A_{12}, A_{23}$ and $A_{13}$ are "independent"? Now check if they are pairwise independent. Also check if they are mutually independent.

SOLUTION: Since the solution is very easy, why not try yourself first, before clicking here? ■

EXAMPLE 4:  Can you give three events $A,B,C$ such that $P(A\cap B\cap C) = P(A)P(B)P(C)$ but still $A,B,C$ are not mutually independent?

SOLUTION: Use the fact $P(\phi)=0$ to get a trivial such example. ■

Product space

We often talk about two random experiments being performed indepndently. For example, tossing a coin and rolling a die in parallel. Let's carefully understand what we mean by this in the case where both the sample spaces are countable (finite/infinite):
Let $\Omega_1$ and $\Omega_2$ be the two sample spaces with corresponding probability functions $P_1,P_2.$

We first combine these to form the Cartesian product $\Omega = \Omega_1\times\Omega_2.$ Notice that $\Omega$ is again countable. For the "coin toss $\times$ die roll" example, this gives $$\Omega = \{(H,1),(H,2),(H,3),(H,4),(H,5),(H,6),(T,1),(T,2),(T,3),(T,4),(T,5),(T,6)\}.$$ Now for each singleton set of the form $\{(a,b)\}$ we define the probability $P(\{(a,b)\}) = P_1(\{a\})\times P_2(\{b\}).$ This uniquely determines $P(A)$ for all $A\subseteq\Omega$ via the probability axioms.

EXAMPLE 5: If the two random experiments are rolling a (fair) die, and these are carried out independently, then the product space may be visualised as 36 points arranged in a square:

Product space of two die rolls
The space is the Cartesian product of the individual spaces. The sample points are the Cartesian products of the individual sample points. But the events in this product space need not always be Cartesian products of events of the two random experiments. Consider the two events shown below:
Two events in the product space
Here $A$ is indeed the Cartesian product of two events (one from each experiment): $$A = \{2,3,4\}\times\{2,3,4,5\}$$. But $B$ cannot be exprssed as a similar Cartesian product. Notice that $A$ looks like a box in the diagram. Indeed, events that can be expressed as Cartesian products of events from the individual random experimnts are sometimes called box events. These play important role when working with infinitely many random experiments. ■

The same construction may be generalised easily for any finite number of random experiments.

Any $A_1\subseteq\Omega_1$ has a natural counterpart in $A\subseteq \Omega$ as $$A =\{(a,b)~:~a\in A_1,~b\in\Omega_2\}.$$ We rarely use different symbols for $A$ and $A_1.$ Just as a real number is considered also to be a complex number. Similarly for any $A_2\subseteq\Omega_2.$

For example, in the "coin toss$\times$ die roll" example, the event $\{$ die shows 3$\}$ becomes $\{(H,3), (T,3)\}.$

With this natural extension, we have the following important theorem about box events.
Theorem Let $\Omega_1,\Omega_2,\Omega,P_1,P_2,P$ be as above. Then for any $A_1\subseteq \Omega_1$ and any $A_2\subseteq \Omega_2$ we have $P(A_1\times A_2) = P_1(A_1)P_2(A_2).$

Proof: The proof is easy if $A_1$ and $A_2$ are both finite.

For the case where at least one of $A_1,A_2$ is infinite (countably infinite as $\Omega_i$'s are countable), we need to use the fact that infinite series of nonnegative terms may be rearranged without changing the value of the series.

[QED]

EXAMPLE 6: I have two dice with probabilities $\left(\frac{1}{16}, \frac{2}{16}, \frac{4}{16}, \frac{3}{16}, \frac{1}{16}, \frac{5}{16}\right)$ and $\left(\frac{2}{12}, \frac{1}{12}, \frac{2}{12}, \frac{1}{12}, \frac{4}{12}, \frac{2}{12}\right)$. These ar rolled independently. What is the probability that both of the show the same number?

SOLUTION: Let $A = $ the event in question.

Then $A = A_1\cup \cdots \cup A_6$, where $A_i = $ the event that both the dice show $i$.

Since a die cannot show two different numbers simulataneously, hence $A_i$'s are disjoint.

Now $P(A_i) = P(\mbox{first die shows }i)\times P(\mbox{second die shows }i)$, since the ice are rolled independently. So $$\begin{eqnarray*} P(A_1) & = & \frac{1}{16}\times\frac{2}{12}\\ P(A_2) & = & \frac{2}{16}\times\frac{1}{12}\\ P(A_3) & = & \frac{4}{16}\times\frac{2}{12}\\ P(A_4) & = & \frac{3}{16}\times\frac{1}{12}\\ P(A_5) & = & \frac{4}{16}\times\frac{1}{12}\\ P(A_6) & = & \frac{2}{16}\times\frac{5}{12}. \end{eqnarray*}$$ Since the $A_i$'s are independent, hence $P(A) = \sum_1^6 P(A_i)$. ■

Infinite product

We shall often talk about an infinite sequence of random experiments, e.g., keep on tossing a coin again and again. Mathematically, we have $\Omega_n$ and $P_n$ for $n=1,2,3,...$ and want to combine them into a single $\Omega.$ Conceptually, this is done much like the finite case. $$\Omega = \Omega_1\times\Omega_2\times\cdots,$$ which means each element of $\Omega$ is a sequence $(a_1,a_2,...),$ where $a_n\in\Omega_n.$ Any $A_n\subseteq \Omega_n$ is lifted to a subset of $\Omega$ as $\{(a_1,...)~:~a_n\in A_n~,~\forall i\neq n~~a_i\in\Omega_i\}.$ Then we want a probability $P$ on $\Omega$ such that for any $A_{n_i}\subseteq \Omega_{n_i}$ for $i=1,...,k$ (for any $k\in{\mathbb N}$) we have them mutually independent under $P.$ Such a $P$ indeed exists and is unique. Unfortunately, this is rather technical to prove.

A few of the difficulties are: In this course, we shall assume the existence and uniqueness of infinite product spaces.

EXAMPLE 7:  A coin with $P(head)=p\in(0,1)$ is tossed again and again independently. Show that we must get a head eventually.

SOLUTION: Here we are working over an infinite product space. For any finite sequence of $H$'s and $T$'s of length $n$ with exactly $k$ many $H$'s, the probability is $p^k(1-p)^{n-k}.$

Let $A_n$ be the event that the first $n$ tosses have produced no $H.$ Also let $A$ denote the event that no $H$ ever occurs.

Then clearly $A_1\supseteq A_2\supseteq A_3\supseteq\cdots$ and $A = \cap A_n.$

Thus $A_n\searrow A.$

Hence $P(A_n)\rightarrow P(A).$

Now $P(A_n) = (1-p)^n\rightarrow 0$ since $p\in (0,1).$

So $P(A)=0.$ Hence $P(A^c) = 1,$ as required. ■

Problems for practice

    EXERCISE 3: (Easy) If $A,B$ are independent, then show that $A^c,B$ are also independent. Are $A^c,B^c$ also independent?

    EXERCISE 4: (Easy) Is it possible to have an event that is independent of itself? If so, find all such events. If not, prove why not.

    EXERCISE 5: (Easy) Is it possible to have an event that is independent of all other events? If so, find all such events. If not, prove why not.

    EXERCISE 6: (Medium) If $A,B$ are independent, and $A,C$ are also independent, then is it true that $A$ and $B\cup C$ must also be independent?

    EXERCISE 7: (Easy) If $A$ and $B$ are mutually exclusive, then must $A,B$ be independent? Must they be dependent?

    EXERCISE 8: (Easy) The numbers $-10,...,-1,1,...,10$ are written on 20 pieces of papers. One of the papers is drawn at random. Let $A$ be the event that the selected number is negative, and $B$ be the event that the selected number has absolute value $>5.$ Are $A,B$ independent?

    EXERCISE 9: (Medium) Same set up as above. Find an event $C$ such that $A,B,C$ are mutually independent.

    EXERCISE 10: (Medium) If $A\subseteq B$ are two events, can $A,B$ be independent?

    EXERCISE 11: (Easy) If $P(A_i)=p_i$ for $i=1,2,3,$ and $A_i$'s are mutually independent, then find $P(A_1\cup A_2\cup A_3).$

    EXERCISE 12: (Easy) If two fair dice are rolled independently, find the probability that the first shows a number strictly larger than the second.

    EXERCISE 13: (Easy) If two fair dice are rolled independently, find the probability that the first number divides the second.

    EXERCISE 14: (Medium)

    Here "in a row" means "consecutively."

    [Hint]

    Let $f = P($Elmer beats father$),$ and $c = P($Elmer beats champion$).$ Then Elmer's chance of winning for the father-champion-father case is $2fc(1-f) +f^2c=A,$ say.

    The winning probability for the other case is: $2cf(1-c) + c^2f=B,$ say.

    We have $f > c.$

    So $A-B < 0.$ Hence Elmer should play champion-father-champion.

    [Corrected a mistake pointed out by Arnab Sanyal.]

    EXERCISE 15: (Medium)

    [Hint]

    Let $A,B$ and $C$, respecively, be the events that the first, second and third man in the three man jury reaches the correct conclusion. Then $P(A)=P(B)=p$ and $P(C)=\frac 12,$ and the three events are independent. Then the event that this jury makes the correct decision is $(A\cap B\cap C^c)\cup(A\cap B^c\cap C)\cup(A^c\cap B\cap C)\cup(A\cap B\cap C).$

    This is $\frac 12(p^2+2p(1-p) +p^2) = p.$

    So both the jury are equally likely to reach the correct decision.

    [Thanks to Shounak Kar for correcting a mistake here.]

    EXERCISE 16: (Easy)

    [Hint]

    $2^{-100}{100 \choose 50}\approx 0.08.$

    EXERCISE 17: (Hard)

    [Hint]

    Deliberately missing (e.g., firing in the air) is also a possible option!