Introduction

De Finetti’s theorem states that an infinite sequence of random variables is exchangeable iff it is a mixture of IID sequences. It is well-known that an approximate statement holds for finite sequences, apart from the details.

We have a look at the simple case of \(2\) rounds, binary outcomes. An exchangeable sequence will have joint probability table \[\begin{array}{l|cc} & 0 & 1 \\ \hline 0 & a & b \\ 1 & b & c \end{array}\] with non-negative \(a\), \(b\) and \(c\) satisfying \(a + 2 b + c = 1\). An IID probability table has just one parameter, and looks like \[\begin{array}{l|cc} & 0 & 1 \\ \hline 0 & (1-p)^2 & p(1-p) \\ 1 & p(1-p) & p^2 \end{array}\] for some \(0 \le p \le 1\). Now any mixture distribution \(\mathbb P\) on \(p\) induces a joint distribution with probability table given by \[\begin{array}{l|cc} & 0 & 1 \\ \hline 0 & \mathop{\mathrm{\mathbb E}}(1-p)^2 & \mathop{\mathrm{\mathbb E}}p(1-p) \\ 1 & \mathop{\mathrm{\mathbb E}}p(1-p) & \mathop{\mathrm{\mathbb E}}p^2 \end{array}\] Writing \(\mu = \mathop{\mathrm{\mathbb E}}[p]\) and \(\sigma^2 = \mathop{\mathrm{\mathbb V}}[p] = \mathop{\mathrm{\mathbb E}}[p^2] - \mu^2\) for the mean and variance of \(\mathbb P\), we can rewrite this table as \[\begin{array}{l|cc} & 0 & 1 \\ \hline 0 & (1-\mu)^2 + \sigma^2 & \mu (1-\mu) - \sigma^2 \\ 1 & \mu (1-\mu) - \sigma^2 & \mu^2 + \sigma^2 \end{array}\] So now the question becomes: for which \(a,b,c\) can we find \(\mu\) and \(\sigma^2\) such that \begin{align*} a &~=~ (1-\mu)^2 + \sigma^2 \\ b &~=~ \mu(1-\mu) - \sigma^2 \\ \mu &~\in~ [0,1] \\ \sigma^2 &~\in~ [0, \mu(1-\mu)] \end{align*} (The last two requirements ensure that \(\mathbb P\) is supported on \([0,1]\). For fixed mean the variance is maximised by the Bernoulli distribution.) Solving the first two equations results in \begin{align*} \mu &~=~ b+c, \\ \sigma^2 &~=~ a-(a+b)^2 . \end{align*} And so the big questions are \(\sigma^2 \le \mu(1-\mu)\), i.e. \[a-(a+b)^2 ~\le~ (b+c)(a+b)\] that is \[a ~\le~ a+b\] which is always true, and \(\sigma^2 \ge 0\), that is \[(a+b)^2 ~\le~ a\] which is equivalent to \[(c + b)^2 ~\le~ c .\] This condition does not hold for all choices of \(a,b,c\) (i.e. for all exchangeable distributions). The subset of the parameter space where it does hold is graphed in the next figure.

Incidentally, the topmost choice \(a=c=0\) and \(b=1/2\), that is, the exchangeable distribution that uniformly plays either \(01\) or \(10\), is furthest from being a mixture of IID. The Bernoulli distributions form the top boundary of the orange region. The bottom boundary is traced out by the mixtures of Bernoulli \(0\) and Bernoulli \(1\).