Allez, un petit billet léger pour bien commencer les vacances de Noël. Toujours sur le projet R de la formation en Data Science pour l’Actuariat, Cyril Legrand a proposé de fusionner deux bases, une sur le revenu moyen par foyer (fiscal) par commune, et une sur le nom de licenciés dans les clubs sportifs. Il … Continue reading Le sport, une activité de riches ?→
There are some graphs that you cannot forget. One graph that I found puzzling was mentioned on Andrew Gelman’s blog, a few years back, and was related to rupture detection What I remember from this graph is that if you want to get a rupture, you can easily find one…
Lors du dernier cours de statistique, nous avons présenter les tests d’ajustment de lois. Nous avions illustré ces tests à partir de la taille d’individus (déjà utilisé pour présenter l’ajustement de lois, et l’estimation de densité), correspondant à . > Davis=read.table( + “http://socserv.socsci.mcmaster.ca/jfox/Books/Applied-Regression-2E/datasets/Davis.txt”) > Davis[12,c(2,3)]=Davis[12,c(3,2)] > X=Davis$height > n=length(X) On notera la statistique d’ordre, au … Continue reading Tests non-paramétriques et simulations→
Juste avant 1850, Adolphe Quételet publiait Sur l’appréciation des documents statistiques, et en particulier sur l’application des moyennes, et introdusait alors le concept d’homme moyen (l’ouvrage est cité dans La théorie des moyennes et son emploi dans les sciences d’observation, de Joseph Lottin, paru en 1909 – je n’ai pas trouvé en ligne cet ouvrage … Continue reading Être – en même temps – moyen et extraordinaire→
A few months ago, I did publish a (long) post entitled ‘some thoughts on economics, mathematics, econometrics, machine learning, etc‘. In that post, I was discussing possible differences between foundations of econometrics, and machine learning. I wanted to get back today on an important point, related to training/sampling datasets, when we have temporal data. I was … Continue reading Econometrics vs. Machine Learning with Temporal Patterns→
Très joli billet sur blog.revolutionanalytics.com avec un code de @kyle_e_walker permettant, très simplement (moyennant une inscription pour avoir une clé permettant d’utiliser l’API du census) de construire une pyramide des âges dynamiques. > devtools::install_github(‘walkerke/idbr’) > library(idbr) > library(ggplot2) > library(animation) > library(dplyr) > library(ggthemes) > idb_api_key(“mykey1239F2f324zf9GGZgege32R2ii4”) On importe alors les données pour les hommes et les femmes, > … Continue reading Dynamique de la Pyramide des Ages→
Following the course of this morning, I got a very interesting question from a student of mine. The question was about having non-significant components in a splineregression. Should we consider a model with a small number of knots and all components significant, or one with a (much) larger number of knots, and a lot of knots non-significant? … Continue reading Regression with Splines: Should we care about Non-Significant Components?→
I think that over the past months, I have been saying non-correct things about classification with categorical covariates. Because I never took time to look at it carefuly. Consider some simulated dataset, with a logistic regression, > n=1e3 > set.seed(1) > X1=runif(n) > q=quantile(X1,(0:26)/26) > q[1]=0 > X2=cut(X1,q,labels=LETTERS[1:26]) > p=exp(-.1+qnorm(2*(abs(.5-X1))))/(1+exp(-.1+qnorm(2*(abs(.5-X1))))) > Y=rbinom(n,size=1,p) > df=data.frame(X1=X1,X2=X2,p=p,Y=Y) Here, we … Continue reading How Could Classification Trees Be So Fast on Categorical Variables?→
Tuesday, at the end of my 5-hour crash course on machine learning for actuaries, Pierre asked me an interesting question about computational time of different techniques. I’ve been presenting the philosophy of various algorithm, but I forgot to mention computational time. I wanted to try several classification algorithms on the dataset used to illustrate the … Continue reading Computational Time of Predictive Models→
Last week, in our mathematical statistics course, we’ve seen the law of large numbers (that was proven in the probability course), claiming that given a collection of i.i.d. random variables, with To visualize that convergence, we can use > m=100 > mean_samples=function(n=10){ + X=matrix(rnorm(n*m),nrow=m,ncol=n) + return(apply(X,1,mean)) + } > B=matrix(NA,100,20) > for(i in 1:20){ + B[,i]=mean_samples(i*10) + } > colnames(B)=as.character(seq(10,200,by=10)) … Continue reading Convergence and Asymptotic Results→
This afternoon, we’ve seen in the training on data science that it was possible to use AIC criteria for model selection. > library(splines) > AIC(glm(dist ~ speed, data=train_cars, family=poisson(link=”log”))) [1] 438.6314 > AIC(glm(dist ~ speed, data=train_cars, family=poisson(link=”identity”))) [1] 436.3997 > AIC(glm(dist ~ bs(speed), data=train_cars, family=poisson(link=”log”))) [1] 425.6434 > AIC(glm(dist ~ bs(speed), data=train_cars, family=poisson(link=”identity”))) [1] 428.7195 … Continue reading Computing AIC on a Validation Sample→
This afternoon, an interesting point was raised, and I wanted to get back on it (since I did publish a post on that same topic a long time ago). How can we adapt a logistic regression when all the observations do not have the same exposure. Here the model is the following: , the occurence of an … Continue reading Modelling Occurence of Events, with some Exposure→
In my previous post, An Attempt to Understand Boosting Algorithm(s), I was puzzled by the boosting convergence when I was using some spline functions (more specifically linear by parts and continuous regression functions). I was using > library(splines) > fit=lm(y~bs(x,degree=1,df=3),data=df) The problem with that spline function is that knots seem to be fixed. The iterative boosting … Continue reading An Update on Boosting with Splines→
Last tuesday, at the annual meeting of the French Economic Association, I was having lunch with Alfred, and while we were chatting about modeling issues (econometric models against machine learning prediction), he asked me what boosting was. Since I could not be very specific, we’ve been looking at wikipedia webpage. Boosting is a machine learning ensemble meta-algorithm for reducing … Continue reading An Attempt to Understand Boosting Algorithm(s)→
A few days ago, I was asked if we should spend a lot of time to choose the distribution we use, in GLMs, for (actuarial) ratemaking. On that topic, I usually claim that the family is not the most important parameter in the regression model. Consider the following dataset > db <- data.frame(x=c(1,2,3,4,5),y=c(1,2,4,2,6)) > plot(db,xlim=c(0,6),ylim=c(-1,8),pch=19) … Continue reading I Fought the (distribution) Law (and the Law did not win)→
"sendo l'intento mio scrivere cosa utile a chi la intende…"