Tag Archives: Tukey

Tukey and Mosteller’s Bulging Rule (and Ladder of Powers)

When discussing transformations in regression models, I usually briefly introduce the Box-Cox transform (see e.g. an old post on that topic) and I also mention local regressions and nonparametric estimators (see e.g. another post). But while I was working on my ACT6420 course (on predictive modeling, which is a VEE for the SOA), I read something about a “Ladder of Powers Rule” also called “Tukey and Mosteller’s Bulging Rule“. To be honest, I never heard about this rule before. But that won’t be the first time I learn something while working on my notes for a course !

The point here is that, in a standard linear regression model, we have

But sometimes, a linear relationship is not appropriate. One idea can be to transform the variable we would like to model, , and to consider

This is what we usually do with the Box-Cox transform. Another idea can be to transform the explanatory variable, , and now, consider,

For instance, this year in the course, we considered – at some point – a continuous piecewise linear functions,

It is also possible to consider some polynomial regression. The “Tukey and Mosteller’s Bulging Rule” is based on the following figure.

and the idea is that it might be interesting to transform  and  at the same time, using some power functions. To be more specific, we will consider some linear model

for some (positive) parameters  and . Depending on the shape of the regression function (the four curves mentioned on the graph above, in the four quadrant) different powers will be considered.

To be more specific, let us generate different models, and let us look at the associate scatterplot,

> fakedataMT=function(p=1,q=1,n=99,s=.1){
+ set.seed(1)
+ X=seq(1/(n+1),1-1/(n+1),length=n)
+ Y=(5+2*X^p+rnorm(n,sd=s))^(1/q)
+ return(data.frame(x=X,y=Y))}
> par(mfrow=c(2,2))
> plot(fakedataMT(p=.5,q=2),main="(p=1/2,q=2)")
> plot(fakedataMT(p=3,q=-5),main="(p=3,q=-5)")
> plot(fakedataMT(p=.5,q=-1),main="(p=1/2,q=-1)")
> plot(fakedataMT(p=3,q=5),main="(p=3,q=5)")

If we consider the South-West part of the graph, to get such a pattern, we can consider

or more generally

where  and  are both larger than 1. And the larger  and/or , the more convex the regression curve.

Let us visualize that double transformation on a dataset, say the cars dataset.

> base=cars
­> names(base)=c("x","y")
> MostellerTukey=function(p=1,q=1){
+ regpq=lm(I(y^q)~I(x^p),data=base)
+ u=seq(min(min(base$x)-2,.1),max(base$x)+2,length=501)
+ par(mfrow=c(1,2))
+ plot(base$x,base$y,xlab="X",ylab="Y",col="white")
+ vic=predict(regpq,newdata=data.frame(x=u),interval="prediction")
+ vic[vic<=0]=.1
+ polygon(c(u,rev(u)),c(vic[,2],rev(vic[,3]))^(1/q),col="light blue",density=40,border=NA)
+ lines(u,vic[,2]^(1/q),col="blue")
+ lines(u,vic[,3]^(1/q),col="blue")
+ v=predict(regpq,newdata=data.frame(x=u))^(1/q)
+ lines(u,v,col="blue")
+ points(base$x,base$y)
+ 
+ plot(base$x^p,base$y^q,xlab=paste("X^",p,sep=""),ylab=paste("Y^",q,sep=""),col="white")
+ polygon(c(u,rev(u))^p,c(vic[,2],rev(vic[,3])),col="light blue",density=40,border=NA)
+ lines(u^p,vic[,2],col="blue")
+ lines(u^p,vic[,3],col="blue")
+ abline(regpq,col="blue")
+ points(base$x^p,base$y^q)
+ }

For instance, if we call

> MostellerTukey(2,1)

we get the following graph,

On the left, we have the original dataset,  and on the right, the transformed one, , with two possible transformations. Here, we did only consider the square of the speed of the car (and only one component was transformed, here). On that transformed dataset, we run a standard linear regression. We add, here, a confidence tube. And then, we consider the inverse transformation of the prediction. This line is plotted on the left. The problem is that it should not be considered as our optimal prediction, since it is clearly biased because . But quantiles associated with a monotone transformation are the transformed quantiles. So confidence tubes can still be considered as confidence tubes.

Note that here, it could have be possible to consider another transformation, with the same shape, but quite different

> MostellerTukey(1,.5)

Of course, there is no reason to consider a simple power function, and the Box-Cox transform can also be used. The interesting point is that the logarithm can be obtained as a particular case. Furthermore, it is also possible to seek optimal transformations, seen here as a pair of parameters. Consider

> p=.1
> bc=boxcox(y~I(x^p),data=base,lambda=seq(.1,3,by=.1))$y
> for(p in seq(.2,3,by=.1)) bc=cbind(bc,boxcox(y~I(x^p),data=base,lambda=seq(.1,3,by=.1))$y)
> vp=boxcox(y~I(x^p),data=base,lambda=seq(.1,3,by=.1))$x
> vq=seq(.1,3,by=.1)
> library(RColorBrewer)
> blues=colorRampPalette(brewer.pal(9,"Blues"))(100)
> image(vp,vq,bc,col=blues)
> contour(vp,vq,bc,levels=seq(-60,-40,by=1),col="white",add=TRUE)

The darker, the better (here the log-likelihood is considered). The optimal pair is here

> bc=function(a){p=a[1];q=a[2]; as.numeric(-boxcox(y~I(x^p),data=base,lambda=q)$y[50])}
> optim(c(1,1), bc,method="L-BFGS-B",lower=c(0,0),upper=c(3,3))
$par
[1] 0.5758362 0.3541601

$value
[1] 47.27395

and indeed, the model we get is not bad,

Fun, ins’t it?

Box plot, Fisher’s style

In a recent issue of Significance, I discovered an interesting – and amuzing – figure, about some box & beard plot, in Dr Fisher’s casebook: Beard the statistician in his den.

In French, the box plot (introduced by John Tukey, not George Box, as discussed in a previous post) is popular under the name boîte à moustaches (box with a mustache, for a simple translation).

> set.seed(2)
> x=rnorm(500)
> boxplot(x,horizontal=TRUE,axes=FALSE)
> axis(1)

I was wondering if it was possible to reproduce that Fisher’s style of box plot, with a beard (Ronald Fisher was famous for his beard). Technically, it is not complicated,

> boxplot(x,horizontal=TRUE,xlim=c(-1,1.3),axes=FALSE)
> axis(1)
> Q=quantile(x,c(.25,.75))
> y=cut(x[(x>=Q[1])&(x<=Q[2])],seq(Q[1],Q[2],length=11))
> tb=table(y)
> u=seq(Q[1],Q[2],length=11)
> umid=(u[1:10]+u[2:11])/2
> for(i in 1:10) segments(umid[i],1-.2,umid[i],1-.2-tb[i]/20,lwd=3)

Yes, I kept the mustaches here, but ploting a simple box, one could easily draw a goatee, or a chin strip

> rect(Q[1],1-.2,Q[2],1+.2)
> segments(median(x),1-.2,median(x),1+.2,lwd=2)

The only problem, is that between the first and the third quartiles, the distribution is much more ‘uniform‘ than it might look on the figure in Significance. Unless, perhaps, if add more bars on the left of the first quartile, and on the right of the third one,

> du=diff(umid)[1]
> y=cut(x,seq(Q[1]-du*8,Q[2]+du*8,length=11+16))
> tb=table(y)
> u=seq(Q[1]-du*8,Q[2]+du*8,length=11+16)
> umid=(u[1:26]+u[2:27])/2
> for(i in 1:8) segments(umid[i],1,umid[i],1-.2-tb[i]/20,lwd=3)
> for(i in 19:26) segments(umid[i],1,umid[i],1-.2-tb[i]/20,lwd=3)

Of course, we can try to add a smile on that face, below the mustache,

> vu=seq(-2.5,2.5,by=.02)
> vv=dnorm(vu)
> lines(vu,1-vv*4,col="red",lty=2)
> d=density(x,bw=.1)
> lines(d$x,1-d$y*4,col="red")

So far, I am not convinced. I mean, it should be possible to add something on the box plot,  to add more information. But I still wonder what… maybe using colors?

Big data, statistics and computer science

Today, software and hardware together provide far more powerful factories than most statisticians realize, factories that many of today’s most able young people find exciting and worth learning about on their own. Their interest can help us greatly, if statistics starts to make much more nearly adequate use of the computer. However, if we fail to expand our uses, their interest in computers can cost us many of our best recruits, and set us back many years.” John Tukey, The technical tools of statistics, 1965, http://jstor.org/… via http://cm.bell-labs.com/…

https://f.hypotheses.org/wp-content/blogs.dir/253/files/2013/02/102646212-05-04.jpeg

Le mois des (boites à) moustaches

On célèbre cette année les 35 ans d’un livre qui a révolutionné la statistique, tant sur les aspects computationnels que graphiques: Exploratory Data Analysis par John Tukey. La légende prétend (on pourra relire a brief history of S par Richard Becker) que ce livre a inspiré ce qui allait devenir R,
Commercial software did not fit well into our research environment. It often used a ‘‘shotgun’’ approach — print out everything that might be relevant to the problem at hand because it could be several hours before the statistician could get another set of output. This was reasonable when computing was done in a batch mode. However, we wanted to be able to interact with our data, using Exploratory Data Analysis (Tukey, 1971) techniques. In addition, commercial statistical software usually didn’t compute what we wanted and was not set up to be modified.

En particulier, c’est dans ce livre qu’est introduit un outil graphique intéressant pour visualiser des données, le boxplot, appelé en français boite à moustaches (que tous les statisticiens se doivent d’utiliser en ce mois de Movembre). Dans la version originale, les boites sont verticales

mais mis horizontalement, on voit une boite, avec des moustaches. Cela dit, les moustaches ne sont pas toujours ce que l’on croit. Par exemple, on pense souvent (et moi le premier, il y a encore peu) que la boite à moustaches permettait de visualiser des valeurs extrêmes, voire des quantiles s’il y a trop de données (le maximum peut – en effet – être très très grand)

C’est malheureusement plus compliqué que ca… Afin de mieux comprendre ce qu’on trace, regardons le jeu de données suivant,

> set.seed(1)
> X=rlnorm(99)


(on est ici en échelle logarithmique). Pour commencer, on peut faire une boite à moustache avec une visualisation du minimum (à gauche) et du maximum (à droite)

> boxplot(X,horizontal=TRUE,log="x",range="0")


Et effectivement, on retrouve facilement tous les paramètres: ceux de la boite (en rouge)

> abline(v=median(X),lty=2,col="red")
> abline(v=quantile(X,.25),lty=2,col="red")
> abline(v=quantile(X,.75),lty=2,col="red")

et ceux de la moustache (en bleu)

> abline(v=min(X),lty=2,col="blue")
> abline(v=max(X),lty=2,col="blue")

Mais ce n’est pas la version standard de la boite à moustaches, qui est – sous R – la suivante

> boxplot(X,horizontal=TRUE,log="x")


Si la borne inférieure de la moustache est toujours le minimum (en tous cas ici), pour la partie supérieure… c’est plus compliqué. Malheureusement ce n’est pas un quantile standard (ceux à 90% et 95% sont représentés ci-dessus en mauve). Pour comprendre ce que c’est, il faut regarder le code: ici, on rajoute au quartile supérieur (la partie de droite de la boite) un pourcentage de la distance inter-quartile (la longueur de la boite), et ce pourcentage est le suivant

> M=boxplot(X)$stats
> (M[5]-M[4])/(M[4]-M[2])
[1] 1.350628
> 2*qnorm(.75)
[1] 1.34898

Plusieurs sites ou ouvrages suggèrent d’utiliser 1.5 fois la longueur interquartile, mais sous R, on utilise . Il faut le savoir (je l’ai appris un peu par hasard en faisant une formation sur la régression quantile au R Montréal Group au printemps passé). Et effectivement, on retrouve les bornes de la moustache,

> abline(v=quantile(X,.75)+2*qnorm(3/4)*IQR(X),lty=2,col="blue")
> abline(v=min(X),lty=2,col="blue")

et je laisse les quantiles pour montrer que (malheureusement) ce n’est pas ce qui est utilisé ici,

> abline(v=quantile(X,.9),lty=2,col="purple")
> abline(v=quantile(X,.95),lty=2,col="purple")

Cela dit, comme je l’évoquais lors de la formation, beaucoup de monde laisse planer une réelle ambiguité sur ce qui est représenté par ces boites à moustaches… Et le fait que ce ne soit pas un quantile me gene un peu… Par exemple, ici on avait la boite à moustaches suivante

Si je prends les 25 plus petites observations, et que je les divise par 4, on obtient

La partie de gauche bouge (c’est normal, c’est la partie qu’on a modifiée), et la médiane reste identique, ainsi que la partie droite de la boite (ce qui est normal, ce sont des quantiles au delà de 25%). Mais de manière un peu gênante, la borne supérieure de la moustache s’est ralongée…  C’est normal, car on utilise une distance inter-quartile. Mais on n’a pas touché aux grandes observations, c’est donc gênant de la voir bouger ainsi… non ?