We determine which exchangeable distributions are mixtures of IID in the simple case of two binary outcomes.

# Introduction

De Finetti’s theorem states that an infinite sequence of random variables is exchangeable iff it is a mixture of IID sequences. It is well-known that an approximate statement holds for finite sequences, apart from the details.

We have a look at the simple case of $$2$$ rounds, binary outcomes. An exchangeable sequence will have joint probability table $\begin{array}{l|cc} & 0 & 1 \\ \hline 0 & a & b \\ 1 & b & c \end{array}$ with non-negative $$a$$, $$b$$ and $$c$$ satisfying $$a + 2 b + c = 1$$. An IID probability table has just one parameter, and looks like $\begin{array}{l|cc} & 0 & 1 \\ \hline 0 & (1-p)^2 & p(1-p) \\ 1 & p(1-p) & p^2 \end{array}$ for some $$0 \le p \le 1$$. Now any mixture distribution $${\mathbb P}$$ on $$p$$ induces a joint distribution with probability table given by $\begin{array}{l|cc} & 0 & 1 \\ \hline 0 & \operatorname*{\mathbb E}(1-p)^2 & \operatorname*{\mathbb E}p(1-p) \\ 1 & \operatorname*{\mathbb E}p(1-p) & \operatorname*{\mathbb E}p^2 \end{array}$ Writing $$\mu = \operatorname*{\mathbb E}[p]$$ and $$\sigma^2 = \operatorname*{\mathbb V}[p] = \operatorname*{\mathbb E}[p^2] - \mu^2$$ for the mean and variance of $${\mathbb P}$$, we can rewrite this table as $\begin{array}{l|cc} & 0 & 1 \\ \hline 0 & (1-\mu)^2 + \sigma^2 & \mu (1-\mu) - \sigma^2 \\ 1 & \mu (1-\mu) - \sigma^2 & \mu^2 + \sigma^2 \end{array}$ So now the question becomes: for which $$a,b,c$$ can we find $$\mu$$ and $$\sigma^2$$ such that \begin{align*} a &~=~ (1-\mu)^2 + \sigma^2 \\ b &~=~ \mu(1-\mu) - \sigma^2 \\ \mu &~\in~ [0,1] \\ \sigma^2 &~\in~ [0, \mu(1-\mu)] \end{align*} (The last two requirements ensure that $${\mathbb P}$$ is supported on $$[0,1]$$. For fixed mean the variance is maximised by the Bernoulli distribution.) Solving the first two equations results in \begin{align*} \mu &~=~ b+c, \\ \sigma^2 &~=~ a-(a+b)^2 . \end{align*} And so the big questions are $$\sigma^2 \le \mu(1-\mu)$$, i.e.$a-(a+b)^2 ~\le~ (b+c)(a+b)$ that is $a ~\le~ a+b$ which is always true, and $$\sigma^2 \ge 0$$, that is $(a+b)^2 ~\le~ a$ which is equivalent to $(c + b)^2 ~\le~ c .$ This condition does not hold for all choices of $$a,b,c$$ (i.e. for all exchangeable distributions). The subset of the parameter space where it does hold is graphed in the next figure.

Incidentally, the topmost choice $$a=c=0$$ and $$b=1/2$$, that is, the exchangeable distribution that uniformly plays either $$01$$ or $$10$$, is furthest from being a mixture of IID. The Bernoulli distributions form the top boundary of the orange region. The bottom boundary is traced out by the mixtures of Bernoulli $$0$$ and Bernoulli $$1$$.