# On Hoeffding’s identity

In 1940, Wassily Hoeffding published Masstabinvariante Korrelationstheorie, which was an impressive paper. For those (like me) who unfortunately barely speak German, an English translation could be found in The Collected Works of Wassily Hoeffding, published a few years ago. As I keep saying in my courses about copulas, almost everything was in that paper, by Wassily Hoeffding. For instance, we can see the following graph, of a cumulative distribution function,

What is the difference with a copula? A copula (in dimension 2) is the cumulative distribution function of a random pair with uniform on $[0,1]\times[0,1]$, as defined by Abe Sklar

But Wassily Hoeffding considered a random pair with uniform on $[-1/2,+1/2]\times[-1/2,+1/2]$. But everything else is the same. He can even derive the level curves of the density of the Gaussian copula,

$c(u,v)=\frac{\varphi_r(\Phi^{-1}(u),\Phi^{-1}(v))}{\varphi(\Phi^{-1}(u))\cdot \varphi(\Phi^{-1}(v))}$

> library(mnormt)
> r=.6
> dc=function(u,v) return(
+ as.numeric(dmnorm(cbind(qnorm(u),qnorm(v)),varcov=
+ matrix(c(1,r,r,1),2,2))/dnorm(qnorm(u))/dnorm(qnorm(v))))
> n=500
> vectu=seq(1/n,1-1/n,length=n-1)
> matdc=outer(vectu,vectu,dc)
> contour(vectu,vectu,matdc,levels=
+ c(.325,.944,1.212,1.250,1.290,1.656,3.85),lwd=2)

But another interesting point is that there is the so-called Hoeffding’s equality

$\text{cov}(X,Y)=\int_{\mathbb{R}\times\mathbb{R}} [F_{XY}(x,y) - F_X(x)F_Y(y)]dxdy$

which is interesting, and quite important, actually, to understand that the covariance (or the correlation) can be seen as some ‘distance‘ to the independence. More precisely, observe that

$\text{cov}(X,Y)=\int_{\mathbb{R}\times\mathbb{R}} [F_{XY}(x,y) - F_{X,Y}^\perp(x,y)]dxdy$

where $F_{X,Y}^\perp$ would be the joint cumulative distribution function of some independent variables, with the same marginal distributions.

Of course, it is not exactly a distance, since it can be negative. But still. Now, the thing is that the proof is not trivial. But it is using interesting identities. For instance, in 1885, Franklin wrote a nice paper, Proof of a Theorem of Tchebycheff’s on Definite Integrals, in the American Journal of Mathematics. To get some heuristics about the identity, consider some (finite) sequences $(\boldsymbol{u})=\{u_1,\cdots,u_n\}$ and $(\boldsymbol{v})=\{v_1,\cdots,v_n\}$, then one can prove that

$\sum_{i,j=1}^n [(u_i-u_j)(v_i-v_j)]=2\left[n\sum_{i=1}^n u_iv_i -\sum_{i=1}^n u_i\cdot \sum_{i=1}^n v_i\right]$

And there is a continuous version of that identity. Consider two bounded functions $u(\cdot)$ and $v(\cdot)$, on some interval, $[a,b]$ then

$\int_{[a.b]\times[a,b]}[(u(x)-u(y))\cdot(v(x)-v(y))]dxdy$

is equal to

$2\left[(b-a)\int_{[a,b]} u(x)v(x)dx -\int_{[a,b]} u(x)dx\cdot \int_{[a,b]} v(x)dx\right]$

In 1979, in Monotone Regression and Covariance Structure, Gerald Shea gave a more probabilistic interpretation of that results, using the fact that

$(b-a)=\int_{[a,b]}1dx$

and using a different measure. More precisely, assume now that  functions $u(\cdot)$ and $v(\cdot)$ are integrable, with respect to some measure $\mu$, on some set $\mathcal{S}\subset\mathbb{R}$. Then

$\int_{\mathcal{S}\times\mathcal{S}}[(u(x)-u(y))\cdot(v(x)-v(y))]\mu(dx)\mu(dy)$

is equal to

$2\left[\mu(\mathcal{S})\int_{\mathcal{S}} u(x)\cdotv(x)\mu(dx) -\int_{\mathcal{S}} u(x)\mu(dx)\cdot \int_{\mathcal{S}} v(x)\mu(dx)\right]$

In the case where $\mu$ is a probability measure of $\mathcal{S}\subset\mathbb{R}$, i.e. $\mu(\mathcal{S})=1$, this equality is the one used by Wassily Hoeffding, in 1940. The interpretation in terms of random variable is simple that

$\mathbb{E}[(X_1-X_2)\cdot(Y_1-Y_2)]=2\left[\mathbb{E}(X_1Y_1)-\mathbb{E}(X_1)\mathbb{E}(Y_1)\right]$

(with standard assuptions of existence of those quantitites) where $(X_1,Y_1)$ and $(X_2,Y_2)$ are two independent vectors, with identical distribution, $F_{XY}$. Actually, this relationship can also be found in Some Concepts of Dependence, by E. L. Lehmann, published in 1966. Oh, and by the way, the connection with Chebyshev inequality (claimed in the title of seminal paper by Franklin) come from the fact that if $u(\cdot)$ and $v(\cdot)$ are monotonic, then the left part of the identity is positive, and thus,

$(b-a)\int_{[a,b]} u(x)\cdot v(x)dx \geq\int_{[a,b]} u(x)dx\cdot \int_{[a,b]} v(x)dx$

But let’s get back to Hoeffiding’s result. How do we get it from that lemma. The idea is to write

$\mathbb{E}[(X_1-X_2)(Y_1-Y_2)]$

as

$\mathbb{E}\left(\int_{\mathbb{R}\times\mathbb{R}} [\boldsymbol{1}_{u\leq X_1}-\boldsymbol{1}_{u\leq X_2}]\cdot[\boldsymbol{1}_{v\leq Y_1}-\boldsymbol{1}_{v\leq Y_2}] dudv\right)$

i.e.

$\mathbb{E}\left(\int_{\mathbb{R}\times\mathbb{R}}[\boldsymbol{1}_{u\leq X_1}\boldsymbol{1}_{u\leq Y_1}-\boldsymbol{1}_{u\leq X_1}\boldsymbol{1}_{u\leq Y_2}-\boldsymbol{1}_{u\leq X_2}\boldsymbol{1}_{u\leq Y_1}+\boldsymbol{1}_{u\leq X_2}\boldsymbol{1}_{u\leq Y_2}]dudv\right)$

We can then intervert the integral and the expectation, use the fact that

$\mathbb{E}(\boldsymbol{1}_{u\leq X_1})=\mathbb{P}(X_1\geq u)$

and then, and some integral calculus can be used to rewrite that expression as

$2\left[\int_{\mathbb{R}\times\mathbb{R}}[F_{X_1Y_1}(u,v)-F_{X_1}(u)F_{Y_1}(v)]dudv\right]$

So we get here Hoeffding’s identity. Actually, as mentioned by Ben Derrett about the equality above, it can be observed (see http://math.stackexchange.com/105713) that$$2\text{cov}(X,Y)=2\big(\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]\big)$$can also be written

$\mathbb E[X_1Y_1]-\mathbb E[X_2]\mathbb E[Y_1]+\mathbb E[X_2Y_2]-\mathbb E[X_1]\mathbb E[Y_2]$

where again, $(X_1,Y_1)$ and $(X_2,Y_2)$ are two independent vectors, with identical distribution, $F_{XY}$. The later can be writen

$\mathbb E((X_1-X_2)(Y_1-Y_2))$