# On Cochran Theorem (and Orthogonal Projections)

Cochran Theorem – from The distribution of quadratic forms in a normal system, with applications to the analysis of covariance published in 1934 – is probably the most import one in a regression course. It is an application of a nice result on quadratic forms of Gaussian vectors. More precisely, we can prove that if $\boldsymbol{Y}\sim\mathcal{N}(\boldsymbol{0},\mathbb{I}_d)$ is a random vector with $d$ $\mathcal{N}(0,1)$ variable then (i) if $A$ is a (squared) idempotent matrix $\boldsymbol{Y}^\top A\boldsymbol{Y}\sim\chi^2_r$ where $r$ is the rank of matrix $A$, and (ii) conversely, if $\boldsymbol{Y}^\top A\boldsymbol{Y}\sim\chi^2_r$ then $A$ is an idempotent matrix of rank $r$. And just in case, $A$ is an idempotent matrix means that $A^2=A$, and a lot of results can be derived (for instance on the eigenvalues). The prof of that result (at least the (i) part) is nice: we diagonlize matrix $A$, so that $A=P\Delta P^\top$, with $P$ orthonormal. Since $A$ is an idempotent matrix observe that$$A^2=P\Delta P^\top=P\Delta P^\top=P\Delta^2 P^\top$$where $\Delta$ is some diagonal matrix such that $\Delta^2=\Delta$, so terms on the diagonal of $\Delta$ are either $0$ or $1$‘s. And because the rank of $A$ (and $\Delta$) is $r$ then there should be $r$ $1$‘s and $d-r$ $1$‘s. Now write$$\boldsymbol{Y}^\top A\boldsymbol{Y}=\boldsymbol{Y}^\top P\Delta P^\top\boldsymbol{Y}=\boldsymbol{Z}^\top \Delta\boldsymbol{Z}$$where $\boldsymbol{Z}=P^\top\boldsymbol{Y}$ that satisfies$\boldsymbol{Z}\sim\mathcal{N}(\boldsymbol{0},PP^\top)$ i.e. $\boldsymbol{Z}\sim\mathcal{N}(\boldsymbol{0},\mathbb{I}_d)$. Thus $$\boldsymbol{Z}^\top \Delta\boldsymbol{Z}=\sum_{i:\Delta_{i,i}-1}Z_i^2\sim\chi^2_r$$Nice, isn’t it. And there is more (that will be strongly connected actually to Cochran theorem). Let $A=A_1+\dots+A_k$, then the two following statements are equivalent (i) $A$ is idempotent and $\text{rank}(A)=\text{rank}(A_1)+\dots+\text{rank}(A_k)$ (ii) $A_i$‘s are idempotents, $A_iA_j=0$ for all $i\neq j$.

Now, let us talk about projections. Let $\boldsymbol{y}$ be a vector in $\mathbb{R}^n$. Its projection on the space $\mathcal V(\boldsymbol{v}_1,\dots,\boldsymbol{v}_p)$ (generated by those $p$ vectors) is the vector $\hat{\boldsymbol{y}}=\boldsymbol{V} \hat{\boldsymbol{a}}$ that minimizes $\|\boldsymbol{y} -\boldsymbol{V} \boldsymbol{a}\|$ (in $\boldsymbol{a}$). The solution is$$\hat{\boldsymbol{a}}=( \boldsymbol{V}^\top \boldsymbol{V})^{-1} \boldsymbol{V}^\top \boldsymbol{y} \text{ and } \hat{\boldsymbol{y}} = \boldsymbol{V} \hat{\boldsymbol{a}}$$
Matrix $P=\boldsymbol{V} ( \boldsymbol{V}^\top \boldsymbol{V})^{-1} \boldsymbol{V}^\top$ is the orthogonal projection on $\{\boldsymbol{v}_1,\dots,\boldsymbol{v}_p\}$ and $\hat{\boldsymbol{y}} = P\boldsymbol{y}$.

Now we can recall Cochran theorem. Let $\boldsymbol{Y}\sim\mathcal{N}(\boldsymbol{\mu},\sigma^2\mathbb{I}_d)$ for some $\sigma>0$ and $\boldsymbol{\mu}$. Consider sub-vector orthogonal spaces $F_1,\dots,F_m$, with dimension $d_i$. Let $P_{F_i}$ be the orthogonal projection matrix on $F_i$, then (i) vectors $P_{F_1}\boldsymbol{X},\dots,P_{F_m}\boldsymbol{X}$ are independent, with respective distribution $\mathcal{N}(P_{F_i}\boldsymbol{\mu},\sigma^2\mathbb{I}_{d_i})$ and (ii) random variables $\|P_{F_i}(\boldsymbol{X}-\boldsymbol{\mu})\|^2/\sigma^2$ are independent and $\chi^2_{d_i}$ distributed.

We can try to visualize those results. For instance, the orthogonal projection of a random vector has a Gaussian distribution. Consider a two-dimensional Gaussian vector

library(mnormt) r = .7 s1 = 1 s2 = 1 Sig = matrix(c(s1^2,r*s1*s2,r*s1*s2,s2^2),2,2) Sig Y = rmnorm(n = 1000,mean=c(0,0),varcov = Sig) plot(Y,cex=.6) vu = seq(-4,4,length=101) vz = outer(vu,vu,function (x,y) dmnorm(cbind(x,y), mean=c(0,0), varcov = Sig)) contour(vu,vu,vz,add=TRUE,col='blue') abline(a=0,b=2,col="red")

Consider now the projection of points $\boldsymbol{y}=(y_1,y_2)$ on the straight linear with directional vector $\overrightarrow{\boldsymbol{u}}$ with slope $a$ (say $a=2$). To get the projected point $\boldsymbol{x}=(x_1,x_2)$ recall that $x_2=ay_1$ and $\overrightarrow{\boldsymbol{x},\boldsymbol{y}}\perp\overrightarrow{\boldsymbol{u}}$. Hence, the following code will give us the orthogonal projections

p = function(a){ x0=(Y[,1]+a*Y[,2])/(1+a^2) y0=a*x0 cbind(x0,y0) }

with

P = p(2) for(i in 1:20) segments(Y[i,1],Y[i,2],P[i,1],P[i,2],lwd=4,col="red") points(P[,1],P[,2],col="red",cex=.7)

Now, if we look at the distribution of points on that line, we get… a Gaussian distribution, as expected,

z = sqrt(P[,1]^2+P[,2]^2)*c(-1,+1)[(P[,1]>0)*1+1] vu = seq(-6,6,length=601) vv = dnorm(vu,mean(z),sd(z)) hist(z,probability = TRUE,breaks = seq(-4,4,by=.25)) lines(vu,vv,col="red")

Or course, we can use the matrix representation to get the projection on $\overrightarrow{\boldsymbol{u}}$, or a normalized version of that vector actually

a=2 U = c(1,a)/sqrt(a^2+1) U [1] 0.4472136 0.8944272 matP = U %*% solve(t(U) %*% U) %*% t(U) matP %*% Y[1,] [,1] [1,] -0.1120555 [2,] -0.2241110 P[1,] x0 y0 -0.1120555 -0.2241110 

(which is consistent with our manual computation). Now, in Cochran theorem, we start with independent random variables,

Y = rmnorm(n = 1000,mean=c(0,0),varcov = diag(c(1,1)))

Then we consider the projection on $\overrightarrow{\boldsymbol{u}}$ and $\overrightarrow{\boldsymbol{v}}=\overrightarrow{\boldsymbol{u}}^\perp$

U = c(1,a)/sqrt(a^2+1) matP1 = U %*% solve(t(U) %*% U) %*% t(U) P1 = Y %*% matP1 z1 = sqrt(P1[,1]^2+P1[,2]^2)*c(-1,+1)[(P1[,1]>0)*1+1] V = c(a,-1)/sqrt(a^2+1) matP2 = V %*% solve(t(V) %*% V) %*% t(V) P2 = Y %*% matP2 z2 = sqrt(P2[,1]^2+P2[,2]^2)*c(-1,+1)[(P2[,1]>0)*1+1]

We can plot those two projections

plot(z1,z2)

and observe that the two are indeed, independent Gaussian variables. And (of course) there squared norms are $\chi^2_{1}$ distributed.