In order to visualize this problem, consider my (simple) dataset
MYOCARDE=read.table(
"http://freakonometrics.free.fr/saporta.csv",
head=TRUE,sep=";")
Let us generate 100 training samples (where we keep about 50% of the observations). On each of them, we use a stepwise procedure, and we keep the estimates of the remaining variables (and their standard deviation actually)
n=nrow(MYOCARDE)
M=matrix(NA,100,ncol(MYOCARDE))
colnames(M)=c("(Intercept)",names(MYOCARDE)[1:7])
S1=S2=M1=M2=M
for(i in 1:100){
idx = which(sample(0:1,size=n, replace=TRUE)==1)
reg=step(glm(PRONO=="DECES"~.,data=MYOCARDE[idx,]))
nm=names(reg$coefficients)
M1[i,nm]=reg$coefficients
S1[i,nm]=summary(reg)$coefficients[,2]
f=paste("PRONO=='DECES'~",paste(nm[1],collapse="+"),sep="")
reg=glm(f,data=MYOCARDE[idx,])
M2[i,nm]=reg$coefficients
S2[i,nm]=summary(reg)$coefficients[,2]
}
Then, for the 7 covariates (and the constant) we can look at the value of the coefficient in the model fitted on the training sample, and the value on the model fitted on the validation sample (of course, only when they were remaining)
for(j in 1:8){
idx=which(!is.na(M1[,j]))
plot(M1[idx,j],M2[idx,j])
abline(a=0,b=1,lty=2,col="gray")
segments(M1[idx,j]2*S1[idx,j],M2[idx,j],M1[idx,j]+2*S1[idx,j],M2[idx,j])
segments(M1[idx,j],M2[idx,j]2*S2[idx,j],M1[idx,j],M2[idx,j]+2*S2[idx,j])
}
For instance, with the intercept, we have the following
where horizontal segments are confidence intervals of the parameter on the model fitted on the training sample, the vertical on the validation sample. The green part means some sort of consistency, while the red one means that actually, the coefficient was negative with one model, positive with the other one. Which is odd (but in that case, observe that coefficients are rarely significant).
We can also visualize the joint distribution of the two estimators,
for(j in 1:8){
library(ks)
idx = which(!is.na(M1[,j]))
Z = cbind(M1[idx,j],M2[idx,j])
H = Hpi(x=Z)
fhat = kde(x=Z, H=H)
image(fhat$eval.points[[1]],
fhat$eval.points[[2]],fhat$estimate)
abline(a=0,b=1,lty=2,col="gray")
abline(v=0,lty=2)
abline(h=0,lty=2)
}
which are here, almost on the diagonal,
meaning that the intercept on the two samples is (more or less) the same. We can then look at other parameters (which is actually more interesting).
On that variable, it seems that it is significant on the training dataset (somehow, it is consistent with the fact that it is remaining in the model after the stepwise procedure) but not on the validation sample (or hardly significant).
Others are much more consistent (with some possible outliers)
On the next one, we have again significance on the training sample, but not on the validation sample,
and probably more interesting
where the two are very consistent.
]]>Sports betting has long fascinated economists and statisticians. Griffith (1949) showed early on that horse race bettors put too much money on horses that have little chance of winning, and too little on those that have the best chance of winning. This tendency to underbid on the most likely event has been obaserved in all sports betting, where the “most likely event” is calculated on the basis of recent statistics. And it can be explained in a fundamental way by the mechanics of mutual betting: the bettor opposes his beliefs to those of the crowd, because the various bets will determine the odds.
Today, in the months leading up to each election, we find ourselves drowned under the polls, conducted every day (and commented on several times a day, as if estimation noise was worth exegesis). As Frédéric Dabi (Deputy Director General of Ifop) pointed out in a debate organised by Risques magazine in 2017, “surveys are an indication of the electoral balance of power, not a prediction”, but it is nevertheless often in the idea of having a prediction that they are used.
But if we go back in time, Rhode & Strumpf (2008) reminds us that other techniques were used, before the idea of surveys became necessary, in particular betting. In 1549, Matteo Dandolo (Ambassador of Veneto) noted that “it is therefore more than clear that the traders are very well informed of the state of the election, and that the employees of the cardinals in conclave (i conclavisti) participate with them in betting, which therefore leads to several tens of thousands of crowns changing hands” as Baumgartner (2003) tells us. Closer to home, betting markets during the elections were popular in the United States until the Second World War. Rhode & Strumpf (2008) suggests several reasons for the loss of interest in the second half of the 20th century: improvements in sampling techniques… and the legalization of horse betting. But online betting sites have revived interest in betting, whatever it may be. Because the sites we mentioned in a previous article are often not limited to sports betting, but also allow betting on a magnitude earthquake, an Oscar winner, or even the observation of the Higgs boson, as proposed by intrade.com, which was liquidated in 2015. As onlinebettingsites.com shows, we could bet on the French elections in 2017, or on the referendum on Brexit (even if for the latter, the predictive markets were not able to reflect the beliefs of the crowds, as an article in The Economist recalls).
Mathématiques du parimutuel
The “parimutual” theory is not unlike the mutualisation of risks, an important foundation of the insurance mechanism, dear to actuaries. Working in the horse betting markets, Edmund Eisenberg and David Gale obtained, in a short threepage article, Consensus of Subjective Probabilities, relatively general results, as long as the bet is static.
Supposons que I joueurs puissent parier sur J chevaux. Chaque joueur possède une somme totale b_{i}, que l’on normalisera de telle sorte que b_{i} désigne la part de la somme totale misée (et donc b_{1} +…+ b_{I} =1). Le joueur i peut alors miser la somme b_{i,j} sur le cheval j (avec ici b_{i,}_{1}+…+b_{i,J} = b_{i}). Lorsque les paris sont clôturés, on note p_{j} le montant parié sur le cheval j, autrement dit b_{1}_{,j}+…+b_{I,j} = p_{j}). La contrainte de budget impose que la somme de ces montants soit égale a 1, ce qui donne aux p_{j} une interprétation probabiliste. Nous reviendrons sur l’utilisation de ces « prix » par la suite. On peut aussi noter q_{j} la cote de gain (payoffodds) définie comme p_{j}^{1}1, de telle sorte que p_{j}=(1+ q_{j})^{ 1}. Si on suppose qu’une portion 1a est gardée par le bookmaker, alors p_{j}= a(1+ q_{j})^{ 1} et q_{j} =( a p_{j})/ p_{j}. Les rendements espérés sur chacun des chevaux doivent être égaux, à l’équilibre, au rendement net attendu, où l’espérance est calculée sous la probabilité p, de manière à refléter les croyances de tous les parieur, soit ici
p_{j}q_{j}+ (1p_{j})(1)= a1
The key result of the Eisenberg & Gale model is to show that there is a balance in this market. More precisely, the fraction bet on each horse must be equal to the probability of the horse market. To achieve this balance, it is often assumed that the equilibrium ratings are found by an auctioneer (this role will be played by the bookmaker). As Blough (2008) noted, the hypothesis that no wagering is made until the odds are balanced is a hypothesis that is indeed true in horse racing.
If we assume that each bettor is risk neutral (and seeks to maximize his expectation of winning) and that his beliefs are materialized by a probability vectors p_{i}=(p_{i}_{1},…,p_{iJ}) – in the sense that player i thinks that horse j will win with a probability p_{ij} – then at equilibrium, if b_{i,j} >0,
p_{ij}=p_{j} max{p_{is}/p_{s}}
where argmax{p_{is}/p_{s}}= argmax{p_{is}(q_{s}+1)}
s the horse on which bettor i must bet everything if he bets on a single horse. Blough (2008) elaborates at length on the description of this balance, and extends it to the case where agents potentially have risk aversion (but the same) and potentially different beliefs. This balance is then described as a consensus of belief.
In an article entitled Interpreting the Predictions of Prediction Markets, Charles Manski proposed using this theory to interpret these prices, in conjunction with more traditional approaches in economics, such as ArrowDebreu prices.
To illustrate this consensus, let us consider a world cup final that should lead either to the victory of A or the victory of B. Let us imagine a contract offering $1 if A wins, and let this contract be offered at price p_{A}. Si on n’autorise pas d’arbitrage, on a une loi du prix unique, et on en déduit que p_{B} = 1p_{A}. Imaginons un joueur qui pense que la probabilité que A gagne est supérieure à p_{A}, soit, avec les notations précédentes, p_{iA} > p_{A}. Dans ce cas, le joueur a intérêt à parier tout son agent sur la victoire de A, c’estàdire acheter ce contrat. La demande agrégée pour ce titre sera alors
[b_{1}P[p_{1A} > p_{A}]+…+ b_{I }P[p_{IA} > p_{A}]] / p_{A}
et on aura un équilibre si la demande agrégée pour les deux titres vérifie
[b_{1}P[p_{1A} > p_{A}]+…+ b_{I }P[p_{IA} > p_{A}]] / p_{A }
= [b_{1}P[p_{1A} < p_{A}]+…+ b_{I }P[p_{IA} < p_{A}]] / p_{B}
de telle sorte que
p_{A} = b_{1}P[p_{1A} > p_{A}]+…+ b_{i}P[p_{iA} > p_{A}]] +…+ b_{I}P[p_{IA} > p_{A}]
which allows the prize to be written as an average of the players’ beliefs.
It should be noted here that the balance is static, allowing the bookmaker to just set a rating. Recently, Agrawal et al (2014) proposed an algorithm to balance this market in continuous time. It may also be noted that this notion of equilibrium appears in many algorithms, such as in the socalled Fisher market.
But this idea of seeing in the prices an aggregation of players’ beliefs is not new! In 1655, in Van Rekeningh in Spelen van Gelucken, (published in Latin under the title’De Ratiociniis in Aleæ Ludo’), Christiaan Huyghens proposed to extract information on beliefs from prices. In 1671, Wilhelmina de Witt noted that, as the price of a contract paying an annuity until death could be seen as a weighted average of annuities (with a fixed maturity), by observing the prices of the different insurance contracts, probabilities interpreted as probabilities of survival could be extracted.
These probabilities are “subjective” as Bruno de Finetti or Frank Ramsey called them. The latter did not see probabilities from a frequentist angle, but saw it as a measure of the degree of belief, which could be measured through bets, in Truth and Probability (1926). This is finally what the theory presented by Kenneth Arrow in 1953, and further developed by Gérard Debreu in 1959, introducing the “ArrowDebreu prices”, says.
Many websites use odds to infer players’ beliefs, which are misrepresented as the probability that a team will win a competition. We can also note the work carried out last summer by doctoral students at the University of Rennes who had compared the odds on online betting sites, and the forecasts obtained by several algorithms (ranging from a naive Bayesian classifier to boosting, SVM or neural networks). A special issue of The Economist, published in 2007, entitled The Future of Futurology, noted that “the most heeded futurists these days are not individuals, but prediction markets, where the informed guesswork of many is consolidated into hard probability”. This idea has now largely returned to the forefront, as predicted in the article by Chen & Pennock (2010) published in AI Magazine.
Agrawal, Shipra, Delage, Erick, Peters, Mark, Wang, Zizhuo & Ye, Yinyu (2014). A Unified Framework for Dynamic Prediction Market Design. Operations Research.
Baron, Ken & Lange, Jeffrey (2006). Parimutuel Applications In Finance: New Markets for New Risks. Springer.
Baumgartner, Frederic (2003) Behind locked doors: a history of papal elections. Palgrave.
Blough, Stephen R. (2008) Differences of opinion at the racetrack. In Efficiency of Racetrack Betting Markets, 323341, World Scientific.
Chen, Yiling & Pennock, David (2010). Designing Markets for Prediction. AI Magazine.
Decker, Wolfgang & Thuillier, JeanPaul (2004). Le sport dans l’antiquité. Picard.
Eisenberg, Edmund & Gale, David (1959). Consensus of Subjective Probabilities: The PariMutuel Method. Annals of Mathematical Statistics, 30:1, 165168.
Griffith, RM (1949) Odds adjustments by American horserace bettors. The American Journal of Psychology, 62, 290294.
Manski, Charles (2005) Interpreting the Predictions of Prediction Markets. NBER 10359.
Rhode, Paul, W. & Strumpf, Koleman (2008) Historical Political Futures Markets: An International Perspective. NBER 14377.
^{[1]} Baron & Lange (2006) discusses the comparison between socalled “riskneutral” valuation in finance (based on the law of single price and arbitrage), and that relating to mutual betting. They thus speak of “selfhedging” because, in a bet, the bettors share the winnings in proportion to their initial bet. This is reminiscent of the way mutual insurance companies operate, where the money used to compensate victims must correspond to the total premiums charged.
]]>
Next week, I will be in Santa Barbara, California. On Monday, I will give a talk on Modeling Risks with Pareto Distributions, at the seminar, and Friday, I will be the first speaker of the InsurTech workshop,
]]>En 1928, revenant sur la révolution chinoise de 1925, André Malraux publie son roman Les Conquérants, et glisse « j’ai appris qu’une vie ne vaut rien, mais que rien ne vaut une vie ». Si la formule peut plaire, on imagine qu’elle n’aidera pas trop un décideur public. En 2013, le Commissariat Général à la stratégie et à la prospective, en France, évaluait la valeur d’une vie à 3 millions d’euros. Mais d’où sort ce chiffre ? Et que signifietil vraiment ?
Chiffrer une vie est en effet un problème auxquels les assureurs, mais aussi les décideurs publics sont confrontés bien plus souvent qu’il n’y paraît. Après l’effondrement du World Trade Center en 2011, le Congrès américain adopta la Loi sur la sécurité du transport aérien et la stabilisation des systèmes. Cette nouvelle loi prévoyait la création d’un fonds spécial pour indemniser les victimes des attentats du 11 septembre 2001. Le montant de l’indemnité, et les personnes qui y auraient droit, seraient décidées par un fonctionnaire toutpuissant. Ce « special master,» Kenneth Feinberg, explique dans un ouvrage qui revient sur son expérience (Feinberg 2006) que le gouvernement souhaitait éviter une avalanche de poursuites pour préjudices corporels. Cela aurait pu plonger l’industrie du transport aérien dans la tourmente.
Un cadre très strict fut donc instauré : seules les victimes « ayant recu un traitement à l’hopital dans les 72 heures qui suivirent les attaques », blessées aux abords du World Trade Center et du Pengatone, ainsi que leurs conjoints et enfants – mais pas leurs parents – furent déclarés admissibles à une indemnisation. Le fond accorda plus de 7 milliards de dollars à 5 560 victimes et membres de leur famille. Feinberg se devait, légalement, d’étalonner les dommages et intérêts en fonction de la « valeur financière » de la victime décédée. Il dû ainsi expliquer à la femme d’un pompier, par exemple, que son mari valait moins qu’un courtier d’assurance.
En France, le récent passage aux 80km/h sur les routes à deux voies fut aussi partiellement justifié par les vies sauvées. Alors que le premier ministre se réjouissait, en janvier, d’un bilan de 116 vies épargnées, la journaliste Alba Ventura (2019) s’interroge sur RTL : « s’il s’agit de ne sauver qu’une vie, estce que ça ne vaut pas le coup ? » Le support radiophonique ajoute à l’ambigüité. Demandetelle en fait si « cela ne vaut pas le coût? » Car le problème est bien celui des méthodes employées par la puissance publique pour chiffrer le prix d’une vie, sauvée ou perdue.
A la fin des années 1940, l’US Air Force cherchait à maximiser les dommages infligés par ses raids aériens contre l’Union Soviétique. Quand un groupe de chercheurs de la RAND Corporation proposa de faire voler un grand nombre d’avions peu coûteux pour leurrer les défenses aériennes soviétiques, les généraux de l’Air Force refusèrent l’idée, arguant que le coût de la vie des pilotes sacrifiés ne figurait pas dans les calculs. Comme le rappelle Spencer Banzhaf (2014), l’économiste de la défense Jack Hirshleifer choisit alors de calculer valeur de la vie d’un pilote en intégrant le coût de sa formation et de son remplacement. Cette réponse avait l’avantage d’utiliser des grandeurs directement monétaires, et facilement quantifiables. Dans les années 1960, sur l’influence des réflexions autour du capital humain, il fut suggéré d’utiliser une estimation des salaires nets actualisés perçus au cours d’une vie de pilote, supposés refléter l’utilité matérielle du métier. Ces méthodes restaient dans la lignée de celles définies par Louis Dublin et Alfred Lotka pour les compagnies d’assurance dans l’entredeuxguerres (Cavalin 2016).
S’il reprend comme titre le slogan publicitaire d’une compagnie d’assurance, popularisé par des organismes de sécurité routière ( « The Life You Save May Be Your Own »),Thomas Schelling, prix Nobel d’économie en 2005, publie en 1968 un article qui rompt largement avec cette tradition. Il utilise en fait le travail de l’un de ses étudiants (et ancien pilote militaire) Jack Carlson, qui cherchait à évaluer si lcertains investissements en matière de sécurité (pour les pilotes) « valaient le coût ». Le coût d’un système d’éjection des avions B58 était par exemple de l’ordre de 80 000 dollars, pour un gain substantiel sur la probabilité de survie. C’est cette idée de lier la valeur de la vie avec la notion de risque qui permis à Schelling de développer le concept de « valeur statistique » de la vie.
L’innovation majeur de Schelling consistait à impliquer les citoyen.ne.s dans l’évaluation de la valeur de leur propre vie. Puisqu’il était stérile de leur demander de chiffrer leur propre vie de but en blanc, on pouvait en révanche adapter la méthode de Carlson en leur demandant, par exemple, combien ils et elles seraient prêtes à dépenser pour un airbag, ou un traitement médical, qui diminuerait leur taux de mortalité de 1%. Ainsi, en se plaçant dans un diagramme représentant en abscisse la probabilité de survie où l’espérance de vie résiduelle et en ordonnée la richesse, comme sur la Figure 1, pouvaiton construire des courbes d’indifférence liant richesse et survie : quelle somme accepteton (marginalement) de dépenser pour gagner statistiquement un peu de vie, soit en diminuant sa probabilité de décès, soit en allongeant son espérance de vie ? Dans l’exemple cidessous, la valeur de la vie est simplement la dérivée de la courbe d’indifférence.
Figure 1 : Arbitrage entre espérance de vie et richesse.
La valeur de la vie n’est alors pas une grandeur constante, mais dépend de la situation dans laquelle on se trouve. Aussi,SVL=\frac{d\omega}{d\text{E}}=\frac{d\omega}{d\text{p}}suivant qu’on la calcule par rapport à une variation de l’espérance de vie, ou de la probabilité de décès. Si on a une espérance de vie résiduelle plus ou moins grande (à gauche ou à droite), ou si on est plus ou moins riche (en haut ou en bas), la pente ne sera pas la même. Un exemple classique est celui d’une roulette russe, avec un pistolet a douze chambres. Supposons qu’il y a 3 balles, quel serait le montant que l’on serait prêt à payer pour enlever une balle ? Que deviendrait ce montant s’il y avait 9 balles et qu’on souhaite en enlever plusieurs ? Supposons que la valeur statistique de la vie soit de 3 millions d’euros. Dans le premier cas, la probabilité passe de 3/12 à 2/12, soit dp_1=1/12 (soit une baisse de 1/3). Dans le second cas, pour avoir aussi une baisse de 1/3, il faudrait passer de 9/12 a 6/12, soit dp_2=3/12. Si on suppose que la valeur statistique de la vie est constante, alors d\omega_2/d\omega_1=d\text{p}_2/d\text{p}_1=3, et on devrait être prêt à dépenser 3 fois plus pour une même baisse relative de probabilité de décès.
D’un point de vue heuristique, dans le second cas, on est dans une situation un peu désespérée (on a 3 chances sur 4 de mourir) et donc toute solution est bonne à prendre, quelle que soit son prix ! C’est ce que l’on retrouve au travers de la convexité de la courbe de droite sur la Figure 1 : si ma probabilité de décès est élevée (à droite sur l’axe des abscisses), je suis prêt à dépenser beaucoup, pour un faible gain. Cette manière d’évaluer sa propre vie, proposée par Schelling, est souvent, appelée « gunpoint value.»
Mais cette approche répondelle vraiment à la question de départ ? La vie sauvée par une mesure de sécurité contraignante et couteuse est rarement celle de la personne qui prend une décision. Cette tension est particulièrement visible au sein des débats français sur la mesure de la valeur d’une vie, puisqu’à la différence des EtatsUnis celleci est largement le fait d’ingénieurséconomistes recrutés par l’Etat afin de mettre en place des politiques publiques visant a augmenter le bienêtre des populations. La question de la sécurité routière est à l’origine d’un article fondateur sur le sujet, présenté par deux ingénieurs des ponts et chaussées, Claude Abraham et Jacques Thedie, au colloque annuel de recherche opérationnelle d’AixenProvence en 1960.
Répondant à la question « combien une collectivité doit dépenser pour sauver une vie humaine », ils pointent deux types d’éléments à quantifier. Les éléments « objectifs » de nature « économique, » quantifiable en actualisant les pertes de salaires directes et les pertes de production et de consommation, grâce un raisonnement pragmatique qui mélange capital humain et analyse macroéonomique. Par exemple, un homme de 4145 ans a une valeur de production deux fois supérieure à un homme de 5660 ans, et sa valeur de consommation est 50% plus élevée. Mais la perte d’un homme de plus de 65 ans est en réalité un gain, ce qui montre l’importance d’intégrer les éléments « affectifs.» Puisque leur évaluation est autrement plus difficile, ceuxci s’en remettent à l’estimation faite par les tribunaux en matière d’indemnisation des dommages personnels, en particulier l’attribution d’un praetium doloris.
Comme le relate Daniel Benamouzig (2005), les aspects théoriques, techniques, éthiques et métaphysiques du principe et de la méthode de la quantification d’une valeur de vie présentés par Abraham et Thédié, et en particulier de l’application de telles méthodes dans le champ de la santé, firent l’objet de débats houleux. Ceuxci n’ont d’ailleurs toujours pas fait l’objet de résolution consensuelle. Françoise Favre (1970) note par exemple que l’utilisation du calcul économique de la valeur d’une annéevie pour décider si un dépistage systématique du cancer du col de l’utérus doit être mis en place peut conduire par construction une réponse négative. En effet, la valeur marchande du travail féminin qui sert de base au calcul est largement inférieur à celle du travail masculin, ce qui crée des inégalités de traitement hommesfemmes.
Adoptant un cadre éthique et théorique empruntant au choix social, Jacques Drèze (1962) propose une méthode de calcul alternative plus proche de celle développée par Schelling. Une décision publique doivent se fonder sur les préférences de la collectivité issues de l’agregation des utilités individuelles pour la décision en question. Une solution aux problèmes de mesure et d’incommensurabilités soulevé par Abraham et Thédié consiste a poser la question aux citoyens « combien la collectivité doitelle dépenser, selon vous, pour sauver une vie? » L’utilité de la vie peutêtre calculée en identifiant la disposition individuelle subjective à payer pour prolonger sa vie en écartant un risque déterminé, ajoute Drèze. Celuici conclue que sa méthode conduit à une estimation de la valeur de vie nettement supérieure à celle à laquelle aboutissent ces collègues. La sensibilité des évaluations aux méthodes de calcul, est, aujourd’hui encore, un problème majeur.
Biausque (2011) reprend plusieurs études, afin d’estimer la valeur (statistique) de la vie, face à des risques environnementaux, de sante ou routier que l’on peut résumer dans le Tableau 1.
Environnement  Santé  Trafic routier  
Nb d’études  51  250  65 
Moyenne (€)  2 455 982  2 574 149  4 884 853 
Minimum (€)  24 427  4 450  267 615 
Maximum (€)  7 641 706  22 100 000  17 500 000 
Tableau 1 : source Biausque (2011).
On voit que ces calculs sont complexes, et donnent lieu à des ordres de grandeurs très différents les uns des autres. La variabilité entre individus était évoquée dans Feinberg (2006) qui expliquait qu’il pouvait être économiquement juste à dire que la vie d’un trader de 25 ans “valait plus” qu’un pompier de 45 ans. Mais c’est surtout la variabilité entre les méthodes, que l’on retrouve également dans Hugonnier et al. (2018) qui surprend, et dérange, avec un facteur allant de 1 à 20 suivant la méthode utilisée.
état de santé  Quintile
0%20% 
Quintile
40%60% 
Quintile
80%100% 

Statistique  ‘fair’  4 380 000  4 400 000  7 890 000 
‘very good’  8 800 000  8 830 000  12 135 000  
Gunpoint  ‘fair’  235 000  235 000  422 000 
‘very good’  590 000  590 000  650 000  
Capital humain  250 000  390 000  525 000 
Tableau 2 : source Hugonnier et al. (2018)
Le Tableau 2 reprend la valeur statistique (inspirée de Drèze 1962), celle basée sur des calculs de capital humain, ainsi qu’une « gunpoint value », en fonction du niveau de richesse de la personne qui décède (niveaux de quantiles) et de l’état de santé de la personne (avant son décès).
Ces tableaux montrent à quel point il est difficile d’évaluer la vie de personnes impliquées dans un accident mortel. On essaye d’imaginer la valeur de la vie d’un « individu représentatif » (peut être en fonction de son état de santé, de son âge, de son revenu). Mais comment faire pour attribuer une valeur à une vie qui n’existe pas encore ? Car nombre de décisions prises aujourd’hui impactant aussi les « générations futures », c’estàdire des personnes qui aujourd’hui n’existent pas… Estil possible de donner une valeur à la vie de ces personnes ? Car c’est normalement ce qu’il convient de faire si on veut mettre en place une politique visant à limiter le réchauffement climatique.
References
Abraham, С. & Thedié, J. 1960 Le prix d’une vie humaine dans les décisions économiques. Revue française de Recherche opérationnelle. 16 : 157168
Banzhaf, Spenser H. 2014. Retrospectives: The ColdWar Origins of the Value of Statistical Life. Journal of Economic Perspective, 28 :4, 213226.
Benamouzig, Daniel. 2005. La Santé au miroir de l’Economie. Paris : PUF
Biausque V. 2011, Valeur statistique de la vie humaine : une métaanalyse. OCDE
Cavalin, C. 2016. « La valeur d’une vie statistique : histoire américaine, histoire de la pensée économique. » Incidence 12.
Commissariat général à la stratégie et à la prospective 2013. Éléments pour une révision de la valeur de la vie humaine. http://www.strategie.gouv.fr/
Costa, Dora L. & Kahn Matthew E. 2004. Changes in the value of life, 19401980. Journal of Risk and Uncertainty, 29 :2, 159180
Drèze, Jacques 1962. L’utilité sociale d’une vie humaine. Revue française de recherche opérationnelle 23 : 3 28
Fabre, Françoise. 1970. « Une étude économique de la prévention et du dépistage précoce du cancer du col de l’utérus » Cahiers du Séminaire d’Econometrie 12, 121143
Feinberg, Kenneth R. 2006. What Is Life Worth?: The Inside Story of the 9/11 Fund and Its Effort to Compensate the Victims of September 11th. Public Affairs.
Johansson, PerOlov, 2000. Is there a meaningful definition of the value of statistical life? Journal of Health Economics, 20, 131139
Hugonnier, J., Pelgrin, F. & StAmour, P. 2018. Valuing Life as an Asset, as a Statistic and at Gunpoint. Swiss Finance Institute Research Paper 1827
Lery, Simon 2004. Arbitrages : le prix de la vie. Alternatives Economiques, 223.
Mrozek, Janusz R. & Taylor, Laura O. 2002. What determines the value of life : a meta analysis. Journal of Policy Analysis and Management, 21 :2, 253–270
Schelling, T.C. 1968. ‘The life you save may be your own.’ In Problems in PublicExpenditure Analysis ed. Samuel B. Chase (Washington DC: Brookings Institution) 127–162
Ventura, Alba. 2019. 80km/h : « S’il s’agit de ne sauver qu’une vie, estce que ça ne vaut pas le coup ? », RTL, 29 janvier 2019.
]]>A report by the American Gaming Association (May 2017) estimated that between $100 billion and $400 billion was bet each year on an estimated gross income of between $5 billion and $20 billion, just for sports betting. We will return here to a brief history of sports betting, emphasizing the concept of parimutuel betting. We will see, in a second article, the links of this principle with mathematical finance, and insurance.
Sports betting has been around for a long time, even if the origin of the first bet is impossible to date. We can think of the Greeks, inventors of the Olympic Games, where it was not uncommon for spectators to bet among themselves on the winners (Decker & Thuiller, 2004). Closer to home, as Georges Vigarello reminds us, “Under the Ancien Régime, gambling was the subject of a real passion. It takes the form of either betting games or prize games.
The first, bets, are made between people from the same social world, between farmers or between nobles. The second, the prize games, take place during parish celebrations, and show different regional practices, with the struggle in Brittany, or the jump in Provence. We can also think of the confrontations between villages at the soule for example. Among the nobles, prize games are organized for special occasions (birth or wedding). These games were recreational and festive moments.
It was not until the end of the 19th century that gambling became a sport, in line with the hygienist theories of the time. We can think of Baron Pierre de Coubertin, who wanted to “use all the means appropriate to develop our physical qualities to make them serve the collective good” through “sport”. We will find the Baron again in 1887 with the creation of the Union of French Societies of Athletic Sports, the official appearance of the notion of “sport”, replacing that of “game”, as Dietschy & Clastres (2006) points out, noting in passing that this Union is based on amateurism, in reaction against the companies of cycling (from 1860) and walking (around 1870) which resumed the traditions of price and betting games. Around 1890, this union, dedicated to athletics, opened up to other sports (rugby, field hockey, fencing, swimming) which were represented by specialized commissions.
A little earlier, during the Industrial Revolution, horse betting organised by bookmakers developed. These bets were popular in the United Kingdom in the 16th and 17th centuries, but remained reserved for the aristocracy and the landed gentry. And in reality, only horse owners were allowed to bet on the results of these private races, known as “matches”. One of his races, launched by the twelfth Earl of Derby (Edward SmithStanley) around 1870, also left its mark on sporting vocabulary. If these races were originally private, Charles II’s passion for these races made them more ambitious, attracting huge crowds, betting more and more important sums. Innkeepers and pub owners were then major promoters of these races, which encouraged owners to organize the races near their establishments. They then naturally became the first bookmakers, organizing the first steeplechases, a form of race (first created in Ireland) where riders ran from one church tower to another by jumping everything in their path! In 1826, at the stables in Saint Alban, north of London, the idea of horses starting and finishing in the same place was launched, giving rise to modern racecourses.
Betting was not yet regulated and betting on races was based on a credit system. And since gambling near a place where alcohol was available in large quantities can have dramatic consequences, the British government banned gambling in pubs, which led to the opening of betting shops, run by bookmakers, with the adoption of the Gambling Act in 1845. The bookmakers not only played the role of scribes, keeping track of transactions in registers, they also served as arbitrators in betting. The bookmaker has become the intermediary with whom to bet, he receives the bets, but does not bet against the player. The arbitrator does not only act at the end, in the event of a dispute, but above all to make the bet official. Indeed, cash bets were rare, and bookmakers decided whether the items bet had the same value and, if not, what the difference was. One of the players then adds money to a cap. Players put their hands in the hat and remove them, either to agree with the assessment or to indicate their disagreement. This is referred to as “hand in cap”, which refers to the amount of money needed to ensure a fair bet. The word “handicap” was then commonly used in horse betting (to designate disadvantaged participants at the start of a race) and then to have a medical connotation from 1950 onwards.
Thereafter, bookmakers will not lack imagination, introducing cash bets, then offering fixed odds against each horse in a race. Parliament then went backwards with the Suppression of Betting Houses Act in 1853. Credit institutions and games of chance on racetracks were allowed. At the same time, in France, Léon Sari invented the “pari mutuel” in 1857 with Charles de Morny, owner of the MaisonsLaffitte racetracks (which became a building with stands in June 1878). Joseph Oller, who cofounded the MoulinRouge, is the concessionaire. As the Senate report on gambling in France reminds us, the law of June 2, 1891 legalizes betting on horse races and establishes the principle of mutualization. As we will see later, this principle means that bettors play against each other and share the winnings (once the legal levies provided for by law have been made for the benefit of the State and the institution of racing). In mathematical finance, we speak of “selfhedging strategy”. In March 1931, the PMU (“pari mutuel urbain”) was born, and it was not until 1985 that the “sports lotto” arrived.
The “pool” has long referred in England to a game of cards played for collective stakes, drawing its etymology from the French “hen”, or rather from the old French “hen”, referring to a young poultry (we will find the Latin word pulla, de pullus, the “young animal”), but also “booty” or “looting”. Here we find the idea of playing for money. This use can be traced back to 1870 (in the sense of “collective betting”) before becoming a pool during the First World War, and then to designate a group of people sharing skills. As early as 1920, the term “football pool” was coined, as recalled by Forrest (1999).
In Liverpool, England, John Moores founded Littlewoods in 1923, a retail company, before launching mail order sales, while offering football bets. The most famous game was the “Treble Chance”, where players could choose to bet on 10, 11 or 12 football matches for the coming weekend. Anecdotally, as noted by Forrest & Pérez (20013), when a match could not take place (for example because of rain), a panel of experts appointed by Littlewoods had to model the match, and provide a forecast. After the Second World War, in Europe, we will see the arrival of socalled 1X2 formulas where the player must predict whether, in a set of 12 to 15 games, the home team will win (1), lose (2) or draw (X). It can be noted that these “football pools” could refer to any form of parimutuel betting, very strongly resembling a lotto. The main difference is that in the lottery, the draw is supposed to be a pure random process, unlike football matches. And for the players, the difference is significant! In the 1980s, Liverpool was one of the largest private companies in Europe. Before decreasing with the birth of online betting sites….
Now, in addition to the betting companies that still exist in the United Kingdom, the strong point of bookmakers is their online presence. The first sites were created around 1995, for example Intertops, which was based on a law passed by the island nation of Antigua and Barbuda (an officially independent, Commonwealth member country) in 1994, granting licences to companies wishing to provide gambling services over the Internet (subsequently, they obtained licences from the Mohawk territory of Kahnawake in Quebec, or Malta). Betting on sports events has quickly become very popular.
In 2000, Betfair was launched, and revolutionized the industry: Betfair itself did not take customer bets, but rather offered customers to place bets between them. These peertopeer betting was quickly very popular. In 2002, the first live betting was launched, offering bettors the opportunity to bet on a sporting event while it was taking place. Today, on lƒes larger sites, all kinds of sports are available, whether collective (football, basketball) or individual (tennis, boxing), with possibly a competition involving more than two players or teams (athletics, cycling). The player can choose an objective, which can be a final score (1X2 in football), a number of goals scored, etc., then he concludes the bet by choosing the amount he is willing to bet (the bet). On all sites, no less than 20,000 bets are possible every day.
Decker, Wolfgang & Thuillier, JeanPaul (2004). Le sport dans l’antiquité. Picard.
Dietschy, Paul & Clastres, Patrick (2006). Sport, société et culture en France du XIXe siècle à nos jours. Hachette, Carré Histoire.
Forrest, David (1999). The Past and Future of the British Football Pools. Journal of Gambling Studies, 15:2, 161176.
Forrest, David & Pérez, Levi (2013) The Football Pools in The Oxford Handbook of the Economics of Gambling, 147162
Vigarello, Georges (2004) Le sport estil encore un jeu ? Sciences Humaines, no 152.
To be continued…. with a post on how bets, predictions and players’ beliefs are linked.
]]>Let us get back to ROC curves to illustrate those points. Consider a very simple dataset with 10 observations (that is not linearly separable)
x1 = c(.4,.55,.65,.9,.1,.35,.5,.15,.2,.85)
x2 = c(.85,.95,.8,.87,.5,.55,.5,.2,.1,.3)
y = c(1,1,1,1,1,0,0,1,0,0)
df = data.frame(x1=x1,x2=x2,y=as.factor(y))
here we can check that, indeed, it is not separable
plot(x1,x2,col=c("red","blue")[1+y],pch=19)
Consider a logistic regression (the course is on linear models)
reg = glm(y~x1+x2,data=df,family=binomial(link = "logit"))
but any model here can be used… We can use our own function
Y=df$y
S=predict(reg)
roc.curve=function(s,print=FALSE){
Ps=(S>=s)*1
FP=sum((Ps==1)*(Y==0))/sum(Y==0)
TP=sum((Ps==1)*(Y==1))/sum(Y==1)
if(print==TRUE){
print(table(Observed=Y,Predicted=Ps))
}
vect=c(FP,TP)
names(vect)=c("FPR","TPR")
return(vect)
}
or any R package actually
library(ROCR)
perf=performance(prediction(S,Y),"tpr","fpr")
We can plot the two simultaneously here
plot(performance(prediction(S,Y),"tpr","fpr"))
V=Vectorize(roc.curve)(seq(5,5,length=251))
points(V[1,],V[2,])
segments(0,0,1,1,col="light blue")
So our code works just fine, here. Let us consider various strategies that should lead us to the diagonal.
The first one is : everyone has the same probability (say 50%)
S=rep(.5,10)
plot(performance(prediction(S,Y),"tpr","fpr"))
V=Vectorize(roc.curve)(seq(0,1,length=251))
points(V[1,],V[2,])
Indeed, we have the diagonal. But to be honest, we have only two points here : (0,0) and (1,1). Claiming that we have a straight line is not very satisfying… Actually, note that we have this situation whatever the probability we choose
S=rep(.2,10)
plot(performance(prediction(S,Y),"tpr","fpr"))
V=Vectorize(roc.curve)(seq(0,1,length=251))
points(V[1,],V[2,])
We can try another strategy, like “making a prediction by tossing of an unbiased coin“. This is what we obtain
set.seed(1)
S=sample(0:1,size=10,replace=TRUE)
plot(performance(prediction(S,Y),"tpr","fpr"))
V=Vectorize(roc.curve)(seq(0,1,length=251))
points(V[1,],V[2,])
segments(0,0,1,1,col="light blue")
We can also try some sort of “random classifier”, where we choose the score randomly, say uniform on the unit interval
set.seed(1)
S=runif(10)
plot(performance(prediction(S,Y),"tpr","fpr"))
V=Vectorize(roc.curve)(seq(0,1,length=251))
points(V[1,],V[2,])
segments(0,0,1,1,col="light blue")
Let us try to go further on that one. For convenience, let us consider another function to plot the ROC curve
V=Vectorize(roc.curve)(seq(0,1,length=251))
roc_curve=Vectorize(function(x) max(V[2,which(V[1,]<=x)]))
We have the same line as previously
x=seq(0,1,by=.025)
y=roc_curve(x)
lines(x,y,type="s",col="red")
But now, consider many scoring strategies, all randomly chosen
MY=matrix(NA,500,length(y))
for(i in 1:500){
S=runif(10)
V=Vectorize(roc.curve)(seq(0,1,length=251))
MY[i,]=roc_curve(x)
}
plot(performance(prediction(S,df$y),"tpr","fpr"),col="white")
for(i in 1:500){
lines(x,MY[i,],col=rgb(0,0,1,.3),type="s")
}
lines(c(0,x),c(0,apply(MY,2,mean)),col="red",type="s",lwd=3)
segments(0,0,1,1,col="light blue")
The red line is the average of all random classifiers. It is not a straight line, be we observe oscillations around the diagonal.
Consider a dataset with more observations
myocarde = read.table("http://freakonometrics.free.fr/myocarde.csv",head=TRUE, sep=";")
myocarde$PRONO = (myocarde$PRONO=="SURVIE")*1
reg = glm(PRONO~.,data=myocarde,family=binomial(link = "logit"))
Y=myocarde$PRONO
S=predict(reg)
plot(performance(prediction(S,Y),"tpr","fpr"))
V=Vectorize(roc.curve)(seq(5,5,length=251))
points(V[1,],V[2,])
segments(0,0,1,1,col="light blue")
Here is a “random classifier” where we draw scores randomly on the unit interval
S=runif(nrow(myocarde)
plot(performance(prediction(S,Y),"tpr","fpr"))
V=Vectorize(roc.curve)(seq(5,5,length=251))
points(V[1,],V[2,])
segments(0,0,1,1,col="light blue")
And if we do that 500 times, we obtain, on average
MY=matrix(NA,500,length(y))
for(i in 1:500){
S=runif(length(Y))
V=Vectorize(roc.curve)(seq(0,1,length=251))
MY[i,]=roc_curve(x)
}
plot(performance(prediction(S,Y),"tpr","fpr"),col="white")
for(i in 1:500){
lines(x,MY[i,],col=rgb(0,0,1,.3),type="s")
}
lines(c(0,x),c(0,apply(MY,2,mean)),col="red",type="s",lwd=3)
segments(0,0,1,1,col="light blue")
So, it looks like me might say that the diagonal is what we have, on average, when drawing randomly scores on the unit interval…
I did mention that an interesting visual tool could be related to the use of the Kolmogorov Smirnov statistic on classifiers. We can plot the two empirical cumulative distribution functions of the scores, given the response Y
score=data.frame(yobs=Y,
ypred=predict(reg,type="response"))
f0=c(0,sort(score$ypred[score$yobs==0]),1)
f1=c(0,sort(score$ypred[score$yobs==1]),1)
plot(f0,(0:(length(f0)1))/(length(f0)1),col="red",type="s",lwd=2,xlim=0:1)
lines(f1,(0:(length(f1)1))/(length(f1)1),col="blue",type="s",lwd=2)
we can also look at the distribution of the score, with the histogram (or density estimates)
S=score$ypred
hist(S[Y==0],col=rgb(1,0,0,.2),
probability=TRUE,breaks=(0:10)/10,border="white")
hist(S[Y==1],col=rgb(0,0,1,.2),
probability=TRUE,breaks=(0:10)/10,border="white",add=TRUE)
lines(density(S[Y==0]),col="red",lwd=2,xlim=c(0,1))
lines(density(S[Y==1]),col="blue",lwd=2)
The underlying idea is the following : we do have a “perfect classifier” (top left corner)
is the supports of the scores do not overlap
otherwise, we should have errors. That the case below
we in 10% of the cases, we might have misclassification
or even more missclassification, with overlapping supports
Now, we have the diagonal
when the two conditional distributions of the scores are identical
Of course, that only valid when n is very large, otherwise, it is only what we observe on average….
]]>By ‘perfect model’ I mean the following : \Omega denotes the heterogeneity factor, because people are different. We would love to get \mathbb{P}[Y=1\Omega]. Unfortunately, \Omega is unobservable ! So we use covariates (like the age of the driver of the car in motor insurance, or of the policyholder in life insurance, etc). Thus, we have data (y_i,\boldsymbol{x}_i)‘s and we use them to train a model, in order to approximate \mathbb{P}[Y=1\boldsymbol{X}]. And then, we check if our model is good (or not) using the ROC curve, obtained from confusion matrices, comparing y_i‘s and \widehat{y}_i‘s where \widehat{y}_i=1 when \mathbb{P}[Y_i=1\boldsymbol{x}_i] exceeds a given threshold. Here, I will not try to construct models. I will predict \widehat{y}_i=1 each time the true underlying probability \mathbb{P}[Y_i=1\omega_i] exceeds a threshold ! The point is that it’s possible to claim a loss (y=1) even if the probability is 3% (and most of the time \widehat{y}=0), and to not claim one (y=0) even if the probability is 97% (and most of the time \widehat{y}=1). That’s the idea with randomness, right ?
So, here p(\omega_1),\cdots,p(\omega_n) denote the probabilities to claim a loss, to die, to fraud, etc. There is heterogeneity here, and this heterogenity can be small, or large. Consider the graph below, to illustrate,
In both cases, there is, on average, 25% chance to claim a loss. But on the left, there is more heterogeneity, more dispersion. To illustrate, I used the arrow, which is a classical 90% interval : 90% of the individuals have a probability to claim a loss in that interval. (here 10%40%), 5% are below 10% (low risk), and 5% are above 40% (high risk). Later on, we will say that we have 25% on average, with a dispersion of 30% (40% minus 10%). On the right, it’s more 25% on average, with a dispersion of of 15%. What I call dispersion is the difference between the 95% and the 5% quantiles.
Consider now some dataset, with Bernoulli variables y, drawn with those probabilities p(\omega). Then, let us assume that we are able to get a perfect model : I do not estimate a model based on some covariates, here, I assume that I know perfectly the probability (which is true, because I did generate those data). More specifically, to generate a vector of probabilities, here I use a Beta distribution with a given mean, and a given variance (to capture the heterogeneity I mentioned above)
a=m*(m*(1m)/v1)
b=(1m)*(m*(1m)/v1)
p=rbeta(n,a,b)
from those probabilities, I generate occurences of claims, or deaths,
Y=rbinom(n,size = 1,prob = p)
Then, I compute the AUC of my “perfect” model,
auc.tmp=performance(prediction(p,Y),"auc")
And then, I will generate many samples, to compute the average value of the AUC. And actually, we can do that for many values of the mean and the variance of the Beta distribution. Here is the code
library(ROCR)
n=1000
ns=200
ab_beta = function(m,inter){
a=uniroot(function(a) qbeta(.95,a,a/ma)qbeta(.05,a,a/ma)inter,
interval=c(.0000001,1000000))$root
b=a/ma
return(c(a,b))
}
Sim_AUC_mean_inter=function(m=.5,i=.05){
V_auc=rep(NA,ns)
b=1
essai = try(ab<ab_beta(m,i),TRUE) if(inherits(essai,what="tryerror")) a=1 if(!inherits(essai,what="tryerror")){ a=ab[1] b=ab[2] } if((a>=0)&(b>=0)){
for(s in 1:ns){
p=rbeta(n,a,b)
Y=rbinom(n,size = 1,prob = p)
auc.tmp=performance(prediction(p,Y),"auc")
V_auc[s]=as.numeric(auc.tmp@y.values)}
L=list(moy_beta=m,
var_beat=v,
q05=qbeta(.05,a,b),
q95=qbeta(.95,a,b),
moy_AUC=mean(V_auc),
sd_AUC=sd(V_auc),
q05_AUC=quantile(V_auc,.05),
q95_AUC=quantile(V_auc,.95))
return(L)}
if((a<0)(b<0)){return(list(moy_AUC=NA))}}
Vm=seq(.025,.975,by=.025)
Vi=seq(.01,.5,by=.01)
V=outer(X = Vm,Y = Vi, Vectorize(function(x,y)
Sim_AUC_mean_inter(x,y)$moy_AUC))
library("RColorBrewer")
image(Vm,Vi,V,
xlab="Probability (Average)",
ylab="Dispersion (Q95Q5)",
col=
colorRampPalette(brewer.pal(n = 9, name = "YlGn"))(101))
contour(Vm,Vi,V,add=TRUE,lwd=2)
On the xaxis, we have the average probability to claim a loss. Of course, there is a symmetry here. And on the yaxis, we have the dispersion : the lower, the less heterogeneity in the portfolio. For instance, with a 30% chance to claim a loss on average, and 20% dispersion (meaning that in the portfolio, 90% of the insured have between 20% and 40% chance to claim a loss, or 15% and 35% chance), we have on average a 60% AUC. With a perfect model ! So with only a few covariates, having 55% should be great !
My point here is that with a low dispersion, we cannot expect to have a great AUC (again, even with a perfect model). In motor insurance, from my experience, 90% of the insured are between 3% chance and 20% chance to claim a loss ! That’s less than 20% dispersion ! and in that case, even if the (average) probability is rather small, it is very difficult to expect an AUC above 60% or 65% !
]]>As explain, the loss is here on an individual basis, so, per policy, the insurer faces the (random) loss S\mathbb{E}[S], which is, on average, null. That’s the second line. For the last line, I keep saying that we look at the overall loss of the insurer, but that’s not correct. Here, with a factor n, we would have the variance of the total loss for the insurance company. We just removed the n factor in the table
That’s what we have below. Here again, on average, the insured should have a null profit. And the total variance (which was \text{Var}[S] in our previous example) is now splitted in two parts (that’s basically Pythagoras theorem).
The interpreration is the following
And then, I usually mention the third and last case, more realistic
And here also, there is a nice interpretation, because of the variance decomposition : there is one part that we observed previously, with some ‘perfect pricing’ and an additional part (that is positive) that is related to the fact that the covariates are just a proxy of the risk factor….
The term on the left is then a lower bound, obtained if actually, using our covariate, available for the pricing, we can get the risk factor.
That was my story, but the fact that n (the portfolio size) was not mentioned in the tables was a bit confusing… So I decided to create some graphs to illustrate those three cases
Consider some simple simulations. On the graph on the left, we have on the xaxis the risk factor, and on the yaxis, the loss (going roughly from 0 to 20). The pure premium is the average of those losses. Here, it’s 10. That’s the plain red line (on the left). In the middle, the yaxis is the insured profit/loss per policy. Someone with a loss close to 0 means a gain of 10, someone with a loss close to 20 means a loss of 10. On average, there is no profit (that’s the plain line). And then, on the right, we have the distribution of the profit/loss (per contract). Again, on average it’s 0, with some variance,
Consider here is simple covariate x : assume here that’s we’ve been able to create a binary variable, that can distinguish the low risks and the high risks. Here, there are two levels for the premium. The low premium is close to 6, and the high one is close to 14. That’s again the graph on the left
Then we have the profit/loss per policy for the insured, in the middle. Here, when the loss was close to 0, the gain is smaller : it is 6 (while it was 10 before). When it was close to 10, previously, it meant a 0 profit, but now it’s either a loss of 4, or a gain of 4. The profit/loss distribution is now on the right. There is less dispersion, and less variance. That the decrease of variance we’ve discussed before. To summarize, segmentation does reduce the variability of the result for the insurance company. That’s what we observe on the right.
Assume now that \Omega is observable. And that we use it for our pricing. The premium is now continuous, and it is the red line, on the left. The profit/loss (in the middle) is the difference between the loss, and its expected value (conditional on the risk factor). And on the right, we have the distribution.
As expected, there is much less variability on the profit/loss distribution of the insurance company in that case. And actually, that’s a lower bound for the variance of result of the insurance company… I hope that the graph clarify what’s going on here…
]]>Consider a linear regression with one single covariate, y=\beta_0+\beta_1x_1+\varepsilon and the leastsquare estimates. The variance of the slope is \text{Var}[\widehat{\beta}_1] Do we decrease this variance if we add one variable, and consider y=\beta_0+\beta_1x_1+\beta_2x_2+\varepsilon ?
For the exam, the expected answer was simply “no”. In a nutshell, there are two cases where we should expect different changes,
We did briefly observed (and discussed) those points on examples during the lecture… but I wanted to go a bit further, since I couldn’t find any analytical results. Let us generate a model y=\beta_0+\beta_1x_1+\beta_2x_2+\varepsilon, and then compare the variance \text{Var}[\widehat{\beta}_1] on the two fitted modes, depending on the correlation between x_1 and x_2
library(mnormt)
n=200
s=function(r=0){
S=matrix(c(1,r,r,1),2,2)
X=rmnorm(n,c(0,0),S)
B=data.frame(y=2+X[,1]+X[,2]+rnorm(n)/2,
x1=X[,1],
x2=X[,2])
reg12=lm(y~x1+x2,data=B)
reg1=lm(y~x1,data=B)
k=summary(reg12)$coefficients[2,2]/summary(reg1)$coefficients[2,2]
k}
Let us generate 500 samples for each value of the correlation, from 0.9 to _0.9
M=NULL
for(r in^{1})
and let us plot the ratio of the two variances
plot(0:1,0:1,xlim=c(1,1),ylim=c(0,2),col="white")
for(i in 1:19) points(rep((((9):9)/10)[i],500),M[,i],col="light blue")
VM=apply(M,2,mean)
lines((((9):9)/10),VM,col="red",lwd=2)
abline(h=1,lty=2)
If the ratio exceeds 1, the variance increases when adding a covariate.
Indeed, here, when the two variables are independent, the variance is divided by two. But when covariates are highly correlated, the variance is multiplied by two…
Now, what if, actually, x_2 is not a real explanatory variable : the true model we generate is y=\beta_0+\beta_1x_1+\varepsilon. In that case,
s=function(r=0){
S=matrix(c(1,r,r,1),2,2)
X=rmnorm(n,c(0,0),S)
B=data.frame(y=2+X[,1]+rnorm(n)/2,
x1=X[,1],
x2=X[,2])
reg12=lm(y~x1+x2,data=B)
reg1=lm(y~x1,data=B)
k=summary(reg12)$coefficients[2,2]/summary(reg1)$coefficients[2,2]
k}
we get our samples as previously
M=NULL
for(r in^{2})
and we plot those ratios
plot(0:1,0:1,xlim=c(1,1),ylim=c(0,2),col="white")
for(i in 1:19) points(rep((((9):9)/10)[i],500),M[,i],col="light blue")
VM=apply(M,2,mean)
lines((((9):9)/10),VM,col="red",lwd=2)
abline(h=1,lty=2)
In the case we add a useless variable x_2, whatever the correlation with x_1, it will always, on average, increase the variance of \widehat{\beta}_1.
Wednesday, at lunch time, I will give a talk on insurance pricing at the (internal) economics seminar, at UQAM. Slides are now online.
In May, I will also give a seminar on the same topic at the Computational Science seminar…
]]>n=100
df=data.frame(matrix(rnorm(n*n),n,n))
names(df)=c("Y",paste("X",1:99,sep=""))
with one variable of interest y, and 99 features x_j. All of them being (by construction) independent. And we have 100 observations… Consider here the regression on the first k features, and compute R_k^2 of that regression
reg=function(k){
frm=paste("Y~",paste("X",1:k,collapse="+",sep=""))
model=lm(frm,data=df)
summary(model)$adj.r.squared}
Let us see what’s going on…
plot(1:99,Vectorize(reg)(1:99))
(actually, it’s not exactly what we have on the graph…. we have the average obtained over 1,000 samples randomly generated, with 90% confidence bands). Oberve that \mathbb{E}[R^2_k]=k/n, i.e. if we add some pure random noise, we keep increasing the R^2 (up to 1, actually).
Good news, as we’ve seen in the course, the adjusted R^2 – denoted \bar R^2might help. Observe that \mathbb{E}[\barR^2_k]=0, so, in some sense, adding features does not help here…
reg=function(k){
frm=paste("Y~",paste("X",1:k,collapse="+",sep=""))
model=lm(frm,data=df)
summary(model)$r.squared}
plot(1:99,Vectorize(reg)(1:99))
We can actually do the same with Akaike criteria AIC_k and Schwarz (bayesian) criteria BIC_k.
reg=function(k){
frm=paste("Y~",paste("X",1:k,collapse="+",sep=""))
model=lm(frm,data=df)
AIC(model)}
plot(1:99,Vectorize(reg)(1:99))
For the AIC, the intitial increase makes sense : we should not prefer the model with 10 covariates, compared with nothing. The strange thing is the far right behavior : we prefer here 80 random noise features to none ! Which I find hard to interprete… For the BIC the code is simply
reg=function(k){
frm=paste("Y~",paste("X",1:k,collapse="+",sep=""))
model=lm(frm,data=df)
BIC(model)}
plot(1:99,Vectorize(reg)(1:99))
and here also, we have the same pattern, where we prefer a big model with juste pure noise to nothing…
A last one to conclude (or not) : what about the leaveoneout cross validation mean squared error ? More precisely, CV=\frac{1}{n}\sum_{i=1}\widehat{\varepsilon}^2_{i}where \widehat{\varepsilon}^2_{i}=y_i\widehat{y}_{i} where \widehat{y}_{i} is the predicted value obtained with the model is estimated when the ith observation is deleted. One can prove that \widehat{\beta}_{i}=\widehat{\beta}(\mathbf{X}^T\mathbf{X})^{1}\mathbf{x}_i\hat\varepsilon_i(1H_{i,i})^{1}where H is the classical hat matrix, thus\widehat{\varepsilon}_{i}=(1H_{i,i})^{1}\hat\varepsilon_ii.e. we do note have to estimate (at each round) n models
reg=function(k){
frm=paste("Y~",paste("X",1:k,collapse="+",sep=""))
model=lm(frm,data=df)
h=lm.influence(model)$hat/2
mean( (residuals(model)/1h)^2 ))}
plot(1:99,Vectorize(reg)(1:99))
Here, it make sense : adding noisy features yields overfit ! So the mean squared error is decreasing !
That’s all nice, but it might not be very realistic… Here, for my model with only one variable, I just pick one, at random…. In practice, we try to get the “best one”… So a more natural idea would be to order the variables according to their correlations with y,
df=data.frame(matrix(rnorm(n*n),n,n))
df=df[,rev(order(abs(cor(df)[1,])))]
names(df)=c("Y",paste("X",1:99,sep=""))}
and as before, we can plot the evolution of R^2_k as a function of k the number of features considered,
which is increasing, with a higher slope at the beginning… For the \bar R^2_k we might actually prefer a correlated noise to nothing (which makes sense actually). So here since we somehow chose our variables, \bar R^2_k seems to be always positive…
For the AIC_k here also, there is an improvement. Before coming back to the original situation (with about 80 features) and here also, we observe the drop on the far right part of the graph
The BIC_k might like the top three features, but soon, we have a deterioration…. even if here also, we have the drop at the far right (with more than 95 features… for 100 observations).
Finally, observe that here again, our (leaveoneout) crossvalidation has not been mesled by our noisy variables : it is always decreasing !
So it seems that crossvalidation techniques are more robust than the AIC and BIC (even if we mentioned in a previous post connexions between all those concepts) when we have a lot a noisy (nonrelevent) features.
]]>In Thinking, Fast and Slow, Daniel Kahneman discusses at length the importance of stereotypes in understanding many decisionmaking processes. A socalled System 1 is used for quick decisionmaking: it allows us to recognize people and objects, helps us focus our attention, and encourages us to fear spiders. It is based on knowledge stored in memory and accessible without intention, and without effort. It can be contrasted with System 2, which allows for more complex decisionmaking, requiring discipline and sequential reflection. In the first case, our brain uses the stereotypes that govern judgments of representativeness, and uses this heuristic to make decisions. If I cook a fish for friends who have come to eat, I open a bottle of white wine. The cliché “fish goes well with white wine” allows me to make a decision quickly, without having to think about it. Stereotypes are statements about a group that are accepted (at least provisionally) as facts about each member. Whether correct or not, stereotypes are the basic tools for thinking about categories in System 1. But in many cases, a more indepth, more sophisticated reflection – corresponding to System 2 – will make it possible to make a more judicious, even optimal decision. Without choosing any red wine, a pinot noir could perhaps also be perfectly suitable for roasted red mullets.
“To generalize is to be an idiot, to particularize is the alone distinction of merit” wrote William Blake around 1800, annotating speeches by the painter Joshua Reynolds. Stigmatizing an entire population because of a minority in a decisionmaking process is a misleading generalization, often punished by society. Moral punishment, but sometimes also legal (when hiring for example) in a society that tends to be civilized, asking not to draw erroneous conclusions about an individual from the statistics of a group to which he would be attached. But isn’t that what the actuary does every day?
For Schauer (2009), this “generalization“, condemned by William Blake, is probably the actuary’s raison d’être: “to be an actuary is to be a specialist in generalization, and actuaries engage in a form of decisionmaking that is sometimes called actuarial“. If I decide to insure a sports car, I have I am given risky driving characteristics that probably belong to the majority of sports car owners, attributes that I may not share. And as we noted in the introduction, insurance companies, of course, are not the only ones that operate actuarially, according to Schauer’s definition. We all do it, much more often than most of us would probably recognize. We do this when we choose airlines based on their safety record, punctuality or lost luggage. We do this when we associate personal characteristics (a visible tattoo, black or brightly coloured clothing) with behavioural characteristics (such as a propensity for violence) that these personal characteristics would seem to indicate. And we operate in this way when we engage in stereotypes that may be harmless on the basis of nationality, for example by calling French people are rude, or Scots all wear kilts, while at the same time acknowledging that more pernicious stereotypes on the basis of ethnic origin, gender, sexual orientation are too widespread today! As the misconception of the word “prejudice” indicates, many people believe that it is unfair to make individual decisions based on nonuniversal group characteristics. Even if group allocations have a solid statistical basis. Because the big difference between actuarial science and everyday life is that actuaries have to use a large number of observations. On a personal level, I can thus decide not to travel with such an airline anymore because on three trips, I have experienced two bad experiences. Before deciding that travel insurance deserves a higher premium when flying with this company, it takes more than three observations!
In fact, the question is often whether an insurance company’s refusal to provide coverage, or to increase the premiums it charges for the same coverage, is an injustice when it is based on an actuarially justified (but perhaps not universal) generalization. As Leemens (2000) noted, the question was asked of the legislator when insurers observed that Jewish women from Eastern Europe were particularly vulnerable to breast and ovarian cancer. At the end of 2012, the European Court of Justice put an end to all discrimination based on the gender of policyholders: insurers were no longer able to differentiate between insurance product prices according to whether the member was male or female. But the use of age is still allowed. Indeed, age is often an indicator of a possible decrease in vision or hearing, slower reaction time (and increased risk of sudden disability), etc. And although there are many individual variations, the available data provide important empirical justification.
A major criticism of machine learning models is the lack of interpretation. But very often, the validation of econometric models requires a narrative built around stereotypes. And this narrative is essential, as Pearl & Mackenzie (2018) reminds us. Indeed, in the “The Ladder of Causation“, there are three levels. At the first level, we find the notion of association (or correlation), or even conditional probability, which serve as a basis for the constitution of stereotypes: if we observe
P[carries  brushing your teeth] < P[carries  don’t brush your teeth]
brushing teeth will be associated with a decrease in the probability of having carries. It is also the basis for regression methods, which are based on correlations between the variable of interest and others, wrongly called explanatory. In Figure 1, we can see the daily cycling traffic in Helsinki, and the average temperature. We will tend to prefer the one on the left, showing the evolution of the number of cyclists as a function of temperature, suggesting that temperature could explain the number of cyclists, and not the other way around. But the stereotype doesn’t necessarily focus on the causal link: if I see a lot of cyclists passing through the window, I’ll tell myself it must be hot, or at least warm.
Figure 1: Näytä Data – Author’s visualization
The first level answers the question “what if I see…?“(e.g. “what cycling traffic to expect if the temperature reaches 20°C? “) and this task can be perfectly accomplished by a machine. The second level is the one that makes it possible to understand an effect, an intervention. The question is then “what if I do…? “. To use our example, we are trying to understand the importance of brushing our teeth on the appearance of cavities. What if brushing your teeth is more natural for children with good teeth? We see the third level of the scale coming up, asking the question “what if I had done…?“and based on the idea of a counterfactual model. We are no longer content to measure correlations, we will build a model explaining what would happen by making a change in the causal variables: what would really happen if the child who did not brush his teeth began to do so? For Pearl & Mackenzie (2018) a human being (maybe even an actuary) can make these more advanced arguments than a machine can (yet) do. And very often, these causal patterns are stereotyped. As Charpentier & Diago Barry (2015) points out, in epidemiology, researchers have long questioned the explanation to be given to the fact that small babies of smokers have a higher probability of survival than babies of nonsmoking mothers. The intuition that something is wrong comes from prejudices, stereotypes that we have, and that a machine cannot have.
As Antonio & Charpentier (2017) noted, the European “gender directive” has confused many insurers who used gender to construct their rates, as the latter was highly correlated with the frequency of claims. But by introducing telematic data, gender was no longer significant in the regression. Gender has long been used as a proxy to capture an effect that can be observed using telematic data, giving rise to many sexist stereotypes and other stereotypes.
But the stories also make it possible to decide between a false correlation (“spurious correlation“) and a correlation that could be interpreted. In Figure 2, we have life expectancy at birth, a variable that we could try to explain in a pension study context, for example, by French department. On the right, two variables taken at random: the number of licenses of a tennis club, and the number of advertising agencies. Stereotypes are what will allow us to construct a causal graph, allowing us to understand why there is such a strong correlation between these variables and life expectancy.
Figure 2: Life expectancy at birth for men, left. At the centre, number of tennis licenses per 100,000 inhabitants (source FFT). On the right, number of advertising agencies per 100,000 inhabitants (source INSEE, code NAF 7311Z). Visualization of the author.
While William Blake condemned stereotypes by saying “to generalize is to be an idiot“, he also clearly went further, continuing with “to particularize is the alone distinction of merit“. This individualisation is also advocated by more and more insurers, and even desired by many insureds. But as Grace & Terry (2002) pointed out, many policyholders suffer from a significant optimism bias – “if I have an accident, it will not be my fault” – leading them to doubt the insurer’s classification – “I’m less risky than the others“. And morality seems to prove them right, against actuaries. Yet, not only is generality not, in general, unjust, but justice itself can have considerable elements of generality. To the extent that justice is centred on equity and to the extent that equity itself is closely linked to equality, then equity, and therefore justice, can now be seen as itself based on the idea of generality. The just society is not necessarily a society in which each individual is treated as an isolated set of unique attributes, requiring individualized attention. On the contrary, in some cases, the just society is a society in which generality is not only unavoidable, but also necessary for justice itself. And pooling risks together is the natural response in an insurance context. And it might not be such a big deal if that class is not as homogenous at it could be, or as we would have expected it to be…
Antonio, K. & Charpentier, A. (2017). La tarification par genre en assurance, corrélation ou causalité ?. Risques. 110 : 107110.
Charpentier, A. & Diago Barry, A. (2015). Big data : passer d’une analyse de corrélation à une interprétation causale. Risques, 101: 107111.
Grace, J. & Terry, M. (2002). Exploring the Causes of Comparative Optimism. Psychologica Belgica. 42: 65–98
Kahneman, D. (2011).Thinking, Fast and Slow. FSG Eds.
Leemens, T. (2000). Selective Justice, Genetic Discrimination, and Insurance: Should We Single Out Genes in Our Laws? McGill law journal. Revue de droit de McGill 45(2):347412.
Pearl, J. & Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect. Basic Books.
Schauer, F.F. (2009). Profiles, Probabilities, and Stereotypes. Harvard University Press.
]]>In econometrics, (numerical) optimization became omnipresent as soon as we left the Gaussian model. We briefly mentioned it in the section on the exponential family, and the use of the Fisher score (gradient descent) to solve the first order condition \mathbf{X}^T W(\beta)^{1})[y\widehat{y}]=\mathbf{0}. In learning, optimization is the central tool. And it is necessary to have effective optimization algorithms, to solve problems (described previously) of the form: \widehat{\beta}\in\underset{\beta\in\mathbb{R}^p}{\text{argmin}}\left\lbrace\sum_{i=1}^n \ell(y_i,\beta_0+\mathbf{x}^T\beta)+\lambda\Vert\boldsymbol{\beta}\Vert\right\rbraceIn some cases, instead of global optimization, it is sufficient to consider optimization by coordinates (widely studied in Daubechies et al. (2004)). If f:\mathbb{R}^d\rightarrow\mathbf{R} is convex and differentiable, if \mathbf{x} satisfies f(\mathbf{x}+h\boldsymbol{e}_i)\geq f(\mathbf{x}) for any h>0 and i\in\{1,\cdots, d\}then f(\mathbf{x})=\min\{f\}, where \mathbf{e}=(\mathbf{e}_i) is the canonical basis of \mathbb{R}^d. However, this property is not true in the nondifferentiable case. But if we assume that the nondifferentiable part is separable (additively), it becomes true again. More specifically, iff(\mathbf{x})=g(\mathbf{x})+\sum_{i=1}^d h_i(x_i)with\left\lbrace\begin{array}{l}g: \mathbb{R}^d\rightarrow\mathbb{R}\text{ convexdifferentiable}\\h_i: \mathbb{R}\rightarrow\mathbb{R}\text{ convex}\end{array}\right.This was the case for Lasso regression, \beta)\mapsto\ \mathbf{y}\beta_0\mathbf{X}\beta\_{\ell_2 }+\lambda\\beta\_{\ell_1}, as shown by Tsen (2001). Getting back to our initial notations, we can use a coordinate descent algorithm: from an initial value \mathbf{x}^{(0)}, we consider (by iterating)x_j^{(k)}\in\text{argmin}\big\lbrace f(x_1^{(k)},\cdots,x_{k1}^{(k)},x_k,x_{k+1}^{(k1)},\cdots,x_n^{(k1)})\big\rbrace for j=1,2,\cdots,nThese algorithmic problems and numerical issues may seem secondary to econometricians. However, they are essential in automatic learning: a technique is interesting if there is a stable and fast algorithm, which allows to obtain a solution. These optimization techniques can be transposed: for example, this coordinate descent technique can be used in the case of SVM methods (known as “vector support” methods) when the space is not linearly separable, and the classification error must be penalized (we will come back to this technique in the next section).
These techniques seem intellectually interesting, but we have not yet discussed the choice of the penalty parameter \lambda. But this problem is actually more general, because comparing two parameters \widehat{\beta}_{\lambda_1} and \widehat{\beta}_{\lambda_2} is actually comparing two models. In particular, if we use a Lasso method, with different thresholds \lambda, we compare models that do not have the same dimension. Previously, we have addressed the problem of model comparison from an econometric perspective (by penalizing overly complex models). In the learning literature, judging the quality of a model on the data used to construct it does not make it possible to know how the model will behave on new data. This is the socalled “generalization” problem. The traditional approach then consists in separating the sample (size n) into two parts: a part that will be used to train the model (the training database, insample, size m) and a part that will be used to test the model (the testing database, outofsample, size nm). The latter then makes it possible to measure a real predictive risk. Suppose that the data are generated by a linear model y_i=\mathbf{x}_i^T \beta_0+\varepsilon_i where \varepsilon_i are independent and centred law achievements. The empirical quadratic risk insample is here\frac{1}{m}\sum_{i=1}^m\mathbb{E}\big([\mathbf{x}_i^T \widehat{\beta}\mathbf{x}_i^T \beta_0]^2\big)=\mathbb{E}\big([\mathbf{x}_i^T \widehat{\beta}\mathbf{x}_i^T \beta_0]^2\big),for any observation i. Assuming the residuals \varepsilon Gaussian, then we can show that this risk is worth \sigma^2 \text{trace} (\Pi_X)/m is \sigma^2 p/m. On the other hand, the empirical outofsample quadratic risk is here \mathbb{E}\big([\mathbf{x}^T \widehat{\beta}\mathbf{x}^T \beta_0]^2\big) where \mathbf{x} is a new observation, independent of the others. It can be noted that \mathbb{E}\big([\mathbf{x}^T \widehat{\beta}\mathbf{x}^T \beta_0]^2\big\vert \mathbf{x}\big)=\text{Var}\big(\mathbf{x}^T \widehat{\beta}\big\vert \mathbf{x}\big)=\sigma^2\mathbf{x}^T(\mathbf{x}^T\mathbf{x})^{1}\mathbf{x},and by integrating with respect to \mathbf{x}, \mathbb{E}\big([\mathbf{x}^T \widehat{\beta}\mathbf{x}^T\beta_0]^2\big)=\sigma^2\text{trace}\big(\mathbb{E}[\mathbf{x}\mathbf{x}^T]\mathbb{E}\big[(\mathbf{x}^T\mathbf{x})^{1}\big]\big).The expression is then different from that obtained insample, and using the Groves & Rothenberg (1969) increase, we can show that \mathbb{E}\big([\mathbf{x}^T \widehat{\beta}\mathbf{x}^T \beta_0]^2\big) \geq \sigma^2\frac{p}{m}which is pretty intuitive, when we start thinking about it. Except in some simple cases, there is no simple (explicit) formula. Note, however, that if \mathbf{X}\sim\mathcal{N}(0,\sigma^2 \mathbb{I}), then \mathbf{x}^T \mathbf{x} follows a Wishart law, and it can be shown that \mathbb{E}\big([\mathbf{x}^T \widehat{\beta}\mathbf{x}^T \beta_0]^2\big)=\sigma^2\frac{p}{mp1}.If we now look at the empirical version: if \widehat{\beta} is estimated on the first m observations,\widehat{\mathcal{R}}^{~\text{ IS}}=\sum_{i=1}^m [y_i\boldsymbol{x}_i^T\widehat{\boldsymbol{\beta}}]^2\text{ and }\widehat{\mathcal{R}}^{\text{ OS}}=\sum_{i=m+1}^{n} [y_i\boldsymbol{x}_i^T\widehat{\boldsymbol{\beta}}]^2and as Leeb (2008) noted, \widehat{\mathcal{R}}^{\text{IS}}\widehat{\mathcal{R}}^{\text{OS}}\approx 2\cdot\nu where \nu represents the number of degrees of freedom, which is not unlike the penalty used in the Akaike test.
Figure 4 shows the respective evolution of \widehat{\mathcal{R}}^{\text{IS}} and \widehat{\mathcal{R}}^{\text{OS}} according to the complexity of the model (number of degrees in a polynomial regression, number of nodes in splines, etc). The more complex the model, the more \widehat{\mathcal{R}}^{\text{IS}} will decrease (this is the red curve, below). But that’s not what we’re interested in here: we want a model that predicts well on new data (i. e. outofsample). As Figure 4 shows, if the model is too simple, it does not predict well (as it does with insample data). But what we can see is that if the model is too complex, we are in a situation of “overlearning”: the model will start to model the noise. Of course, this figure should remind us of the one we’ve seen in our second post of that series
Figure 4 : Generalization, under and overfitting
Instead of splitting the database in two, with some of the data that will be used to calibrate the model and some to study its performance, it is also possible to use crossvalidation. To present the general idea, we can go back to the “jackknife”, introduced by Quenouille (1949) (and formalized by Quenouille (1956) and Tukey (1958)) relatively used in statistics to reduce bias. Indeed, if we assume that \{y_1,\cdots,y_n\} is a sample drawn according to a law F_\theta, and that we have an estimator T_n (\mathbf{y})=T_n (y_1,\cdots,y_n), but that this estimator is biased, with \mathbf{E}[T_n (\mathbf{Y})]=\theta+O(n^{1}), it is possible to reduce the bias by considering \widetilde{T}_n(\mathbf{y})=\frac{1}{n}\sum_{i=1}^n T_{n1}(\mathbf{y}_{(i)})\text{ where }\mathbf{y}_{(i)}=(y_1,\cdots,y_{i1},y_{i+1},\cdots,y_n)It can then be shown that \mathbb{E}[\tilde{T}_n(Y)]=\theta+O(n^{2})The idea of crossvalidation is based on the idea of building an estimator by removing an observation. Since we want to build a predictive model, we will compare the forecast obtained with the estimated model, and the missing observation\widehat{\mathcal{R}}^{\text{ CV}}=\frac{1}{n}\sum_{i=1}^n \ell(y_i,\widehat{m}_{(i)}(\mathbf{x}_i))We will speak here of the “leaveoneout” (loocv) method.
This technique reminds us of the traditional method used to find the optimal parameter in exponential smoothing methods for time series. In simple smoothing, we will construct a forecast from a time series as {}_t\widehat{y}_{t+1} =\alpha\cdot{}_{t1}\widehat{y}_t +(1\alpha)\cdot y_t, where \alpha\in[0,1], and we will consider as “optimal” \alpha^\star = \underset{\alpha\in[0,1]}{\text{argmin}}\left\lbrace \sum_{t=2}^T \ell({}_{t1}\widehat{y}_{t},y_{t}) \right\rbraceas described by Hyndman et al (2009).
The main problem with the leaveoneout method is that it requires calibration of n models, which can be problematic in large dimensions. An alternative method is cross validation by kblocks (called “kfold cross validation”) which consists in using a partition of \{1,\cdots,n\} in k groups (or blocks) of the same size, \mathcal{I}_1,\cdots,\mathcal{I}_k, and let us note \mathcal{I}_{\bar j}=\{1,\cdots,n\}\setminus \mathcal{I}_j. By noting \widehat{m}_{(j)} built on the sample \mathcal{I}_{\bar j}, we then set:\widehat{\mathcal{R}}^{k\text{ CV}}=\frac{1}{k}\sum_{j=1}^k \mathcal{R}_j\text{ where }\mathcal{R}_j=\frac{k}{n}\sum_{i\in\mathcal{I}_{{j}}} \ell(y_i,\widehat{m}_{(j)}(\mathbf{x}_i))Standard crossvalidation, where only one observation is removed each time (loocv), is a special case, with k=n. Using k=5 or 10 has a double advantage over k=n: (1) the number of estimates to be made is much smaller, 5 or 10 rather than n; (2) the samples used for estimation are less similar and therefore less correlated to each other, which tends to avoid excess variance, as recalled by James et al. (2013).
Another alternative is to use boosted samples. Let \mathcal{I}_b be a sample of size n obtained by drawing with replacement in \{1,\cdots,n\} to know which observations (y_i,\mathbf{x}_i) will be kept in the learning population (at each draw). Note \mathcal{I}_{\bar b}=\{1,\cdots,n\}\setminus\mathcal{I}_b. By noting \widehat{m}_{(b)} built on sample \mathcal{I}_b, we then set :\widehat{\mathcal{R}}^{\text{ B}}=\frac{1}{B}\sum_{b=1}^B \mathcal{R}_b\text{ where }\mathcal{R}_b=\frac{n_{\overline{b}}}{n}\sum_{i\in\mathcal{I}_{\overline{b}}} \ell(y_i,\widehat{m}_{(b)}(\mathbf{x}_i))where n_{\bar b} is the number of observations that have not been kept in \mathcal{I}_b. It should be noted that with this technique, on average e^{1}\sim36.7\% of the observations do not appear in the boosted sample, and we find an order of magnitude of the proportions used when creating a calibration sample, and a test sample. In fact, as Stone (1977) had shown, the minimization of AIC is to be compared to the crossvalidation criterion, and Shao (1997) showed that the minimization of BIC corresponds to kfold crossvalidation, with k=n/\log n.
All those techniques here are mentioned in the “machine learning” section since they rely on automatic, computational techniques, and no probabilistic foundations are necessary. In many cases we did use the notation m^\star (at least in the first posts on “machine learning” techniques) to highlight the fact that we want some sort of “optimal” model – and to make a distinction with estimators \widehat{m} considered earlier, when we had some probabilistic framework. But of course, it is possible (and necessary) to build bridges between those two cultures…
References are online here. As explained in the introduction, it is some sort of online version of an introduction to our joint paper with Emmanuel Flachaire and Antoine Ly, Econometrics and Machine Learning (initially writen in French), that will actually appear soon in the journal Economics and Statistics (in English and in French).
One important concept in econometrics is Ockham’s razor – also known as the law of parsimony (lex parsimoniae) – which can be related to abductive reasoning.
Akaike’s criterion was based on a penalty of likelihood taking into account the complexity of the model (the number of explanatory variables retained). If in econometrics, it is customary to maximize the likelihood (to build an asymptotically unbiased estimator), and to judge the quality of the expost model by penalizing the likelihood, the strategy here will be to penalize exante in the objective function, even if it means building a biased estimator. Typically, we will build: (\widehat{\beta}_{0,\lambda},\widehat{\beta}_{\lambda})=\text{argmin}\left\lbrace\sum_{i=1}^n \ell(y_i,\beta_0+\mathbf{x}^T\beta)+\lambda \text{ penalization}( \boldsymbol{\beta})\right\rbrace, ~~~(11)where the penalty function will often be a norm \\cdot\ chosen a priori, and a penalty parameter \lambda (we find in a way the distinction between AIC and BIC if the penalty function is the complexity of the model – the number of explanatory variables retained). In the case of the \ell_2 norm, we find the ridge estimator, and for the \ell_1 norm, we find the lasso estimator (“Least Absolute Shrinkage and Selection Operator”). The penalty previously used involved the number of degrees of freedom of the model, so it may seem surprising to use \\beta\_{\ell_2} as in the ridge regression. However, we can envisage a Bayesian vision of this penalty. It should be recalled that in a Bayesian model : \underbrace{\mathbb{P}[\boldsymbol{\theta}\vert\boldsymbol{y}]}_{\text{posterior}} \propto \underbrace{\mathbb{P}[\boldsymbol{y}\vert\boldsymbol{\theta}]}_{\text{likelihood}} \cdot \underbrace{\mathbb{P}[\boldsymbol{\theta}]}_{\text{prior}} or\log\mathbb{P}[\boldsymbol{\theta}\vert\boldsymbol{y}]= \underbrace{\log \mathbb{P}[\boldsymbol{y}\vert\boldsymbol{\theta}]}_{\text{log likelihood}} + \underbrace{\log\mathbb{P}[\boldsymbol{\theta}]}_{\text{{penalty}}}In a Gaussian linear model, if we assume that the a priori law of \theta follows a centred Gaussian distribution, we find a penalty based on a quadratic form of the components of \theta.
Before going back in detail to these two estimators, obtained using the \ell_1 or \ell_2 norm, let us return for a moment to a very similar problem: the best choice of explanatory variables. Classically (and this will be even more true in large dimension), we can have a large number of explanatory variables, p, but many are just noise, in the sense that \beta_j=0 for a large number of j. Let s be the number of (really) relevant covariates, s=\#S, with S=\{j=1,\cdots,p:\beta_j\neq 0\}. If we note \mathbf{X}_S the matrix composed of the relevant variables (in columns), then we assume that the real model is of the form y=\mathbf{x}_S^T \beta_S+\varepsilon. Intuitively, an interesting estimator would then be \widehat{\beta}_S=[\mathbf{X}_S^T \mathbf{X}_S ]^{1} \mathbf{X}_S^T \mathbf{y}, but this estimator is only theoretical because the set S is unknown, here. This estimator can actually be seen as the oracle estimator mentioned above. One may then be tempted to solve (\widehat{\beta}_{0,s},\widehat{\beta}_{s})=\underset{\beta_S\in\mathbb{R}^s}{\text{argmin}}\left\lbrace\sum_{i=1}^n \ell(y_i,\beta_0+\mathbf{x}^T\beta_S)\right\rbrace,\text{ s.t. } \# {S}=s This problem was introduced by Foster & George (1994) using the \ell_0 notation. More precisely, let us define here the following three norms, where \mathbf{a}\in\mathbb{R}^d, \Vert\boldsymbol{a} \Vert_{\ell_0}=\sum_{i=1}^d \mathbf{1}(a_i\neq 0), ~~ \Vert\mathbf{a} \Vert_{\ell_1}=\sum_{i=1}^d a_i~~\text{ and }~~\Vert\mathbf{a} \Vert_{\ell_2}=\left(\sum_{i=1}^d a_i^2\right)^{1/2}
Table 1: Constrained optimization and regularization.
Let us consider the optimization problems in Table 1. If we consider the classical problem where the quadratic norm is used for \ell, the two problems of the equation (\ell1) of Table 1 are equivalent, in the sense that, for any solution (\beta^\star,s) to the left problem, there is \lambda^\star such that (\beta^\star,\lambda^\star) is the solution of the right problem; and vice versa. The result is also true for problems(\ell2). These are indeed convex problems. On the other hand, the two problems (\ell0) are not equivalent: if for (\beta^\star,\lambda^\star) solution of the right problem, there is s^\star such that \beta^\star is solution of the left problem, the reverse is not true. More generally, if you want to use an \ell_p norm, sparsity is obtained if p\leq 1 whereas you need p\geq1 to have the convexity of the optimization program.
One may be tempted to resolve the penalized program (\ell0) directly, as suggested by Foster & George (1994). Numerically, it is a complex combinatorial problem in large dimension (Natarajan (1995) notes that it is a NPdifficult problem), but it is possible to show that if \lambda\sim\sigma^2 \log(p), then \mathbb{E}\big([\mathbf{x}^T \widehat{\beta}\mathbf{x}^T \beta_0]^2\big) \leq \underbrace{\mathbb{E}\big(\mathbf{x}_{ {S}}^T\widehat{\beta}_{{S}}\mathbf{x}^T \beta_0]^2\big)}_{=\sigma^2 \#{S}}\cdot \big(4\log p+2+o(1)\big) Observe that in this case \widehat{\beta}_{\lambda,j}^{\text{sub}} = \left\lbrace\begin{array}{l}0 \text{ if } j\notin{S}_\lambda(\beta)\\ \widehat{\beta}_{j}^{\text{ols}} \text{ if } j\in{S}_\lambda(\beta),\end{array}\right. where S_\lambda (\beta) refers to all nonzero coordinates when solving (\ell0).
The problem (\ell2) is strictly convex if \ell is the quadratic norm, in other words, the Ridge estimator is always well defined, with in addition an explicit form for the estimator, \widehat{ {\beta}}_\lambda^{\text{ ridge}}=(\mathbf{X}^T\mathbf{X}+\lambda\mathbb{I})^{1}\mathbf{X}^T\mathbf{y}=(\mathbf{X}^T\mathbf{X}+\lambda\mathbb{I})^{1}(\mathbf{X}^T\mathbf{X})\widehat{ {\beta}}^{\text{ ols}} Therefore, it can be deduced that \text{bias}[\widehat{ {\beta}}_\lambda^{\text{ ridge}}]=\lambda[\mathbf{X}^T\mathbf{X}+\lambda\mathbb{I}]^{1}~\widehat{ {\beta}}^{\text{ ols}} and\text{Var}[\widehat{\beta}_\lambda^{\text{ ridge}}]=\sigma^2[\mathbf{X}^T\mathbf{X}+\lambda\mathbb{I}]^{1}\mathbf{X}^T\mathbf{X}[\mathbf{X}^T\mathbf{X}+\lambda\mathbb{I}]^{1}With a matrix of orthonormal explanatory variables (i.e. \mathbf{X}^T \mathbf{X}=\mathbb{I}), the expressions can be simplified \text{bias}[\widehat{ {\beta}}_\lambda^{\text{ ridge}}]=\frac{\lambda}{1+\lambda}~\widehat{ {\beta}}^{\text{ ols}}\text{ and }\text{Var}[\widehat{ {\beta}}_\lambda^{\text{ ridge}}]=\frac{\sigma^2}{(1+\lambda)^2}\mathbb{I} Observe that \text{Var}[\widehat{ {\beta}}_\lambda^{\text{ ridge}}]<\text{Var}[\widehat{ {\beta}}^{\text{ ols}}]. And because \text{mse}[\widehat{ {\beta}}_\lambda^{\text{ ridge}}]=\frac{p\sigma^2}{(1+\lambda)^2}+\frac{\lambda^2}{(1+\lambda)^2}\beta^T\beta we obtain an optimal value for \lambda: \lambda^\star=k\sigma^2/\beta^T\beta
On the other hand, if \ell is no longer the quadratic norm but the \ell_1 norm, the problem (\ell1) is not always strictly convex, and in particular, the optimum is not always unique (for example if \mathbf{X}^T \mathbf{X} is singular). But if it is strictly convex, then predictions \mathbf{X}\beta will be unique. It should also be noted that two solutions are necessarily consistent in terms of sign of coefficients: it is not possible to have \beta_j<0 for one solution and \beta_j>0 for another. From a heuristic point of view, the program (\ell1) is interesting because it allows to obtain in many cases a corner solution, which corresponds to a problem resolution of type (\ell0) – as shown visually on Figure 2.
Figure 2 : Penalization based on norms \ell_0, \ell_1 and \ell_2 (from Hastie et al. (2016)).
Let us consider a very simple model: y_i=x_i \beta+\varepsilon, with a penalty \ell_1 and a loss function \ell_2. The problem (\ell2) then becomes \min\big\{\mathbf{y}^T\mathbf{y}2\mathbf{y}^T\mathbf{x}\beta+\beta\mathbf{x}^T\mathbf{x}\beta+2\lambda\beta\big\} The first order condition is then 2\mathbf{y}^T\mathbf{x} + 2\mathbf{x}^T\mathbf{x}\widehat{\beta}\pm 2\lambda=0And the sign of the last term depends on the sign of \beta. Suppose that the least square estimator (obtained by setting \lambda=0) is (strictly) positive, i. e. \mathbf{y}^T \mathbf{x}>0. If \lambda is not too big, we can imagine that \beta is of the same sign as \widehat{\beta}^{\text{mco}}, and therefore the condition becomes 2\mathbf{y}^T \mathbf{x}+2\mathbf{x}^T \mathbf{x}\beta+2\lambda=0, and the solution is \widehat{\beta}_{\lambda}^{\text{ lasso}}=\frac{\mathbf{y}^T\mathbf{x}\lambda}{\mathbf{x}^T\mathbf{x}} By increasing \lambda, there will be a time such that \widehat{\beta}_λ=0. If we increase \lambda a bit little more, \widehat{\beta}_λ does not become negative because in this case the last term of the first order condition changes, and in this case we try to solve 2\mathbf{y}^T\mathbf{x} + 2\mathbf{x}^T\mathbf{x}\widehat{\beta} 2\lambda=0 whose solution is then \widehat{\beta}_{\lambda}^{\text{ lasso}}=\frac{\mathbf{y}^T\mathbf{x}+\lambda}{\mathbf{x}^T\mathbf{x}}But this solution is positive (we assumed \mathbf{y}^T \mathbf{x}>0), and so it is possible to have \widehat{\beta}_\lambda <0at the same time. Also, after a while, \widehat{\beta}_\lambda=0, which is then a corner solution. Things are of course more complicated in larger dimensions (Tibshirani & Wasserman (2016) goes back at length on the geometry of the solutions) but as Candès & Plan (2009) notes, under minimal assumptions guaranteeing that the predictors are not strongly correlated, the Lasso obtains a quadratic error almost as good as if we had an oracle providing perfect information on the set of \beta_j‘s that are not zero. With some additional technical hypotheses, it can be shown that this estimator is “sparsistant” in the sense that the support of \widehat{\beta}_\lambda^{\text{lasso}} is that of \beta, in other words Lasso has made it possible to select variables (more discussions on this point can be obtained in Hastie et al. (2016)).
More generally, it can be shown that \widehat{\beta}_\lambda^{\text{lasso}} is a biased estimator, but may be of sufficiently low variance that the mean square error is lower than using least squares. To compare the three techniques, relative to the least square estimator (obtained when \lambda=0), if we assume that the explanatory variables are orthonormal, then \widehat{\beta}_{\lambda,j}^{\text{ subset}}=\widehat{\beta}_{j}^{\text{ ols}}\boldsymbol{1}_{\widehat{\beta}_{\lambda,j}^{\text{ subset}}>b}, ~~\widehat{\beta}_{\lambda,j}^{\text{ ridge}}=\frac{\widehat{\beta}_{j}^{\text{ ols}}}{1+\lambda}and\widehat{\beta}_{\lambda,j}^{\text{ lasso}}=\text{sign}[\widehat{\beta}_{j}^{\text{ ols}}]\cdot(\widehat{\beta}_{j}^{\text{ ols}}\lambda)_+
Figure 3 : Penalization based on norms , and (from Hastie et al. (2016)).
To be continued with probably a final post this week (references are online here)…
]]>As we have seen before, modelling here is based on solving an optimization problem, and solving the problem described by equation (6) is all the more complex because the functional space \mathcal{M} is large. The idea of boosting, as introduced by Shapire & Freund (2012), is to learn, slowly, from the errors of the model, in an iterative way. In the first step, we estimate a model m_1 for y, from \mathbf{X}, which will give an error \varepsilon_1. In the second step, we estimate a model m_2 for \varepsilon_1, from X, which will give an error \varepsilon_2, etc. We will then retain as a model, after k iterations m^{(k)}(\cdot)=\underbrace{m_1(\cdot)}_{\sim y}+\underbrace{m_2(\cdot)}_{\sim \epsilon_1}+\underbrace{m_3(\cdot)}_{\sim \epsilon_2}+\cdots+\underbrace{m_k(\cdot)}_{\sim \epsilon_{k1}}=m^{(k1)}(\cdot)+m_k(\cdot)~~~(7)Here, the error \varepsilon is seen as the difference between y and the model m(\mathbf{x}), but it can also be seen as the gradient associated with the quadratic loss function. Formally, \varepsilon can be seen as \nabla\ell in a more general context (here we find an interpretation that reminds us of residuals in generalized linear models).
Equation (7) can be seen as a descent of the gradient, but written in a dual way. The problem will then be rewritten as an optimization problem: m^{(k)}=m^{(k1)}+\underset{h\in\mathcal{H}}{\text{argmin}}\left\lbrace \sum_{i=1}^n \ell(\underbrace{y_im^{(k1)}(\boldsymbol{x}_i)}_{\varepsilon_{k,i}},h(\boldsymbol{x}_i))\right\rbrace~~~(8)where the trick is to consider a relatively simple space \mathcal{H} (we will speak of “weak learner”). Classically, \mathcal{H} functions are stepfunctions (which will be found in classification and regression trees) called “stumps”. To ensure that learning is indeed slow, it is not uncommon to use a shrinkage parameter, and instead of setting, for example, \varepsilon_1=ym_1 (\mathbf{x}), we will set \varepsilon_1=y\alpha\cdot m_1 (\mathbf{x}) with \alpha\in[0.1]. It should be noted that it is because a nonlinear space is used for \mathcal{H}, and learning is slow, that this algorithm works well. In the case of the Gaussian linear model, remember that the residuals \varepsilon=y\mathbf{x}^T\beta are orthogonal to the explanatory variables, \mathbf{X}, and it is then impossible to learn from our errors. The main difficulty is to stop in time, because after too many iterations, it is no longer the m function that is approximated, but the noise. This problem is called overlearning.
This presentation has the advantage of having a heuristic reminiscent of an econometric model, by iteratively modelling the residuals by a (very) simple model. But this is often not the presentation used in the learning literature, which places more emphasis on an optimization algorithm heuristic (and gradient approximation). The function is learned iteratively, starting from a constant value, m^{(0)}=\underset{m\in\mathbb{R}}{\text{argmin}}\left\lbrace\sum_{i=1}^n \ell(y_i,m)\right\rbracethen we consider the following learning procedure{\displaystyle m^{(k)}=m^{(k1)}+{\underset{h\in {\mathcal {H}}}{\text{argmin}}}\sum _{i=1}^{n}\ell(y_{i},m^{(k1)}(\mathbf{x}_{i})+h(\mathbf{x}_{i}))}~~~(9)which can be written, if \mathcal{H} is a set of differentiable functions, {\displaystyle m^{(k)}=m^{(k1)}\gamma_{k}\sum _{i=1}^{n}\nabla _{m^{(k1)}}\ell(y_{i},m^{(k1)}(\mathbf{x}_{i})),} where {\displaystyle \gamma _{k}=\underset{\gamma }{\text{argmin }}\sum _{i=1}^{n}\ell\left(y_{i},m^{(k1)}( \mathbf{x}_{i})\gamma \nabla _{m^{(k1)}}\ell(y_{i},m^{(k1)}( \mathbf{x}_{i}))\right).} To better understand the relationship with the approach described above, at step k, pseudoresiduals are defined by setting r_{i,k}=\left.\frac{\partial \ell(y_i,m(\mathbf{x}_i))}{\partial m(\mathbf{x}_i)}\right\vert_{m(\mathbf{x})=m^{(k1)}( \mathbf{x})}\text{ where }i=1,\cdots,nA simple model is then sought to explain these pseudoresiduals according to the explanatory variables \mathbf{x}_i, i.e. r_{i,k}=h^\star(\mathbf{x}_i) , where h^\star\in\mathcal{H}. In a second step, we look for an optimal multiplier by solving\gamma_k = \underset{\gamma\in\mathbb{R}}{\text{argmin}}\left\lbrace\sum_{i=1}^n \ell(y_i,m^{(k1)}( \mathbf{x}_i)+\gamma h^\star(\mathbf{x}_i))\right\rbrace then update the model by setting m_k (\cdot)=m_(k1) (\cdot)+\gamma_k h^\star (\cdot) . More formally, we move from equation (8) – which clearly shows that we are building a model on residuals – to equation (9) – which will then be translated as a gradient calculation problem – noting that \ell(y,m+h)=\ell(ym,h) . Classically, class \mathcal{H} of functions consists in regression trees. It is also possible to use a form of penalty by setting m_k (\cdot)=m_(k1) (\cdot)+\nu\gamma_k h^\star (\cdot) , with \nu\in(0,1) . But let’s go back a little further – in our next post – on the importance of penalization before discussing the numerical aspects of optimization.
To be continued (keep in mind that references are online here)…
]]>We have a training sample, with observations (\mathbf{x}_i,y_i) where the variables y are in a set \mathcal{Y}. In the case of classification, \mathcal{Y}=\{1,+1\}, but a relatively general set can be considered (note that if econometricians prefer \mathcal{Y}=\{0,1\} – because of the Bernoulli distribution and because 0 and 1 are lower and upper bounds of probabilities, people in the “machine learning” community prefer \mathcal{Y}=\{1,+1\}). A predictor m is an function taking values in \mathcal{Y}, used to label (or classify) future new observations, using some features that lie in a set \mathcal{X}. It is assumed that the labels are produced by an (unknown) classifier f called target. For a statistician, this function would be the real model. Naturally, we want to build m as close as possible to f. Let \mathbb{P} be a (unknown) distribution on \mathcal{X}. The error of m with respect to target f is defined by \mathcal{R}_{\mathbb{P},f}(m)=\mathbb{P}[m(\boldsymbol{X})\neq f(\boldsymbol{X})]\text{ where }\boldsymbol{X}\sim\mathbb{P}or equivalently,\mathcal{R}_{\mathbb{P},f}(m)=\mathbb{P}\big[\{\boldsymbol{x}\in\mathcal{X}:m(\boldsymbol{x})\neq f(\boldsymbol{x})\}\big]To obtain our “optimal” classifier, it becomes necessary to assume that there is a link between the data in our sample and the pair (\mathbb{P},f) , i.e. a data generation model. We will then assume that the \mathbf{x}_i are obtained by independent draws according to \mathbb{P}, and that then y_i=f(\mathbf{x}_i) . We can define the empirical risk of a classifier m, as \widehat{{R}}(m)=\frac{1}{n}\sum_{i=1}^n \boldsymbol{1}(m(\boldsymbol{x}_i)\neq y_i)
It is important to recognize that a perfect model cannot be found, in the sense that R_{\mathbb{P},f} (m)=0. Indeed, if we consider the simplest case, with \mathcal{X}=\{x_1,x_2\} and \mathbb{P} is such that \mathbb{P}(\{x_1\})=p and \mathbb{P}(\{x_2\})=1p. The probability of never observing \{x_2\} among the n observations is (1p)^n, and if p<1/n, it is quite likely never to observe \{x_2\} so it can never be predicted. We cannot therefore hope to have a zero risk whatever \mathbb{P}. And more generally, it is also possible to observe \{x_1\} and \{x_2\}, and despite everything, to make mistakes on the labels. Also, instead of looking for a perfect model, we can try to have an “approximately correct” model. We will then try to find m such that R_{\mathbb{P},f} (m)\leq\varepsilon, where \varepsilon is an a priori specified threshold. But even this condition is too strong, and cannot be fulfilled. Thus, we will usually as to have R_{\mathbb{P},f} (m)\leq\varepsilon with some probability 1\delta. Hence, we will try to be “probably approximately correct” (PAC), allowing to make a mistake with a probability \delta, again fixed a priori.
Also, when we build a classifier, we do not know either \mathbb{P} or f, but we give ourselves a precision criterion \varepsilon , and a confidence parameter \delta, and we have n observations. Note that n, \varepsilon and \delta can be linked. We then look for a model m such that R_{\mathbb{P},f} (m)\leq\varepsilon with probability (at least) 1\delta, so that we are probably approximately correct. Wolpert (1996) has shown (see details in Wolpert & Macready (1997)) that there is no universal learning algorithm. In particular, it can be shown that there is \mathbb{P} such that R_{\mathbb{P},f} (m) is relatively high, with a relatively high probability (also).
The interpretation is that since we cannot learn (in the PAC sense) about all the functions m, we will then force m to belong to a particular class, noted \mathcal{M}. Let us suppose, to start with, that \mathcal{M} contains a finite number of possible models. We can then show that for all \varepsilon and \delta, that for all \mathbb{P} and f, if we have enough observations (more precisely n\geq \varepsilon^{1} \log[\delta^{1} \mathcal{M}], then with a greater probability than 1\delta, R_{\mathbb{P},f} (m^\star)\leq\varepsilon where m^\star \in \underset{m\in\mathcal{M}}{\text{argmin}}\Big\lbrace\frac{1}{n}\sum_{i=1}^n \boldsymbol{1}(m(\boldsymbol{x}_i)\neq y_i)\Big\rbracein other words m^\star is a model in \mathcal{M} that minimizes empirical risk.
We can go a little further, staying in the case where \mathcal{Y}=\{1,+1\}. An \mathcal{M} class of classifiers will be called PAClearnable if there is n_M:[0,1]^2\rightarrow \mathbb{N} such that, for all \varepsilon, \delta, \mathbb{P} and if it is assumed that the target f belongs to \mathcal{M}, then using n>n_M (\varepsilon,\delta) observations \mathbf{x}_i drawn from \mathbb{P}, labelled y_i by f, then there is m\in\mathcal{M} such that, with probability 1\delta, R_{\mathbb{P},f} (m)\leq\varepsilon. The n_M function is then called “sample complexity to learn”. In particular, we have seen that if M contains a finite number of classifiers, then \mathcal{M} is PAClearnable with complexity n_M (\varepsilon,\delta)=\varepsilon^{1} \log[\delta^{1} M].
Naturally, we would like to have a more general result, especially if \mathcal{M} is not finite. To do this, the VC dimension of VapnikChervonenkis must be used, which is based on the idea of shattering points (for a binary classification). Consider k points \{x_1,\cdot,x_k\}, and consider the set {E}_k=\big\lbrace(m(\boldsymbol{x}_1),\cdots,m(\boldsymbol{x}_k))\text{ for }m\in\mathcal{M})\big\rbrace Note that the elements of E_k belong to \{1,+1\}^k. In other words, E_k \leq 2^k. We will say that M shatter all the points if all the combinations are possible, i. e. E_k =2^k. Intuitively, the labels of the set of points do not provide enough information on target f, because anything is possible. The VC dimension of \mathcal{M} is then VC(\mathcal{M})=\sup\big\lbrace k\text{ such that }\mathcal{M}\text{ shatters }\{\boldsymbol{x}_1,\cdots\boldsymbol{x}_k\}\big\rbrace
For example, if \mathcal{X}=\mathbb{R} and all (simple) models of the form [1] m_{a,b}=\mathbf{1}_{\pm}(x\in[a,b]) are considered. No set of \{x_1,x_2,x_2,x_3\} ordered points can be shattered because it is sufficient to assign respectively +1, 1 and +1 to x_1, x_2 and x_3 respectively, therefore VC<3. On the other hand \{0,1\} is shattered, so VC\geq 2. The dimension of this predictor set is 2: If we increase by one dimension, \mathcal{X}=\mathbb{R}^2 and consider all (simple) models of the form m_{a,b}=\mathbf{1}_{\pm} (x\in[a,b]) (where [a,b] refers to the rectangle), then the dimension of \mathcal{M} is here 4.
To introduce SVMs, let’s place ourselves in the case where \mathcal{X}=\mathbb{R}^k, and consider separations by hyperplanes passing through the origin (we will say homogeneous), in the sense that m_{\mathbf{w}} (\mathbf{x})=\mathbf{1}_{\pm}(\mathbf{w}^T \mathbf{x}\geq 0) . It can be shown that no set of k+1 points can be shattered by these two homogeneous spaces in \mathbb{R}^k, and therefore VC(M)=k. If we add a constant, in the sense that m_{\mathbf{w},b} (\mathbf{x})=\mathbf{1}_{\pm}(\mathbf{w}^T \mathbf{x}+b\geq 0), we can show that no set of k+2 points can be sprayed by these two (nonhomogeneous) spaces in \mathbb{R}^k, and therefore VC(M)=k+1. This dimension reminds us of the dimension of the model we’ve seen in the econometric context.
From this dimension VC, we deduce the socalled fundamental theorem of learning: if \mathcal{M} is a class of dimension d=VC(M) , then there are positive constants \underline{C} and \overline{C} such as the sample complexity for M to be PAClearnable satisfies \underline{C}\epsilon^{1}\big(d+\log[\delta^{1}]\big)\leq n_{\mathcal{M}}(\epsilon,\delta) \leq \overline{C}\epsilon^{1}\big(d\log[\epsilon^{1}]+\log[\delta^{1}]\big)The link between the notion of learning (as defined in Vailiant (1984)) and the VC dimension was clearly established in Blumer et al (1989).
Nevertheless, while the work of Vapnik and Chervonenkis is considered to be the foundation of statistical learning, Thomas Cover’s work in the 1960s and 1970s should also be mentioned, in particular Cover (1965) on the capacities of linear models, and Cover & Hart (1967) on learning in the context of the algorithm of the knearest neighbors. These studies have linked learning, information theory (with the textbook Cover & Thomas (1991)), complexity and statistics. Other authors have subsequently brought the two communities closer together, in terms of learning and statistics. For example, Halbert White proposed to see neural networks in a statistical context in White (1989), going so far as to state that « learning procedures used to train artificial neural networks are inherently statistical techniques. It follows that statistical theory can provide considerable insight into the properties, advantages, and disadvantages of different network learning methods ». This turning point in the late 1980s will anchor learning theory in a probabilistic context.
These choices (of objective and loss function) are essential, and very dependent on the problem under consideration. Let us begin by describing a historically important model, Rosenblatt’s (1958) “perceptron”, introduced into classification problems, where y\in\{1,+1\}, inspired by McCulloch & Pitts (1943). We have data \{(y_i,\mathbf{x}_i)\}, and we will iteratively build a set of m_k[\mathbf{x} models, where at each step, we will learn from the errors of the previous model. In the perceptron, a linear model is considered so that :m(\mathbf{x})=\boldsymbol{1}_{\pm}(\beta_0+\mathbf{x}^T \boldsymbol{\beta}\geq 0)=\left\lbrace\begin{array}{l}+1\text{ si }\beta_0+\mathbf{x}^T \boldsymbol{\beta}\geq 0\\1\text{ si }\beta_0+\mathbf{x}^T \boldsymbol{\beta}< 0\end{array}\right.where \beta coefficients are often interpreted as “weights” assigned to each of the explanatory variables. We give ourselves initial weights (\beta_0^{(0)},\beta^{(0)} , which we will update taking into account the prediction error made, between y_i and the prediction \widehat{y}_i^{(k)} :\widehat{y}_i^{(k)}=m^{(k)}(\mathbf{x}_i)=\boldsymbol{1}_{\pm}(\beta_0^{(k)}+\mathbf{x}^T \boldsymbol{\beta}^{(k)}\geq 0), with, in the case of the perceptron:\beta_j^{(k+1)}={\beta}_j^{(k)}+\eta\underbrace{(\mathbf{y}\widehat{\mathbf{y}}^{(k)})^T}_{=\ell({\mathbf{y}},\widehat{\mathbf{y}}^{(k)})}\mathbf{x}_jHere \ell(y,y')=\mathbf{1}(y\neq y') is a loss function, which will allow to give a price to an error made, by predicting \widehat{y}=m(\mathbf{x}) and observing y. For a regression problem, we can consider a quadratic error \ell_2, such that \ell(y,m(\mathbf{x}))=(ym(\mathbf{x}))^2 or in absolute value \ell_1, with \ell(y,m(\mathbf{x}))=ym(\mathbf{x}). Here, for our classification problem, we used a misqualification indicator (we could discuss the symmetry of this loss function, suggesting that a false positive costs as much as a false negative). Once this loss function has been specified, we recognize in the problem previously described a gradient descent, and we see that we are trying to solve:m^\star(\mathbf{x})=\underset{m\in\mathcal{M}}{\text{argmin}}\left\lbrace\sum_{i=1}^n \ell(y_i,m(\mathbf{x}_i))\right\rbrace~~~(6)for a predefined set of predictors \mathcal{M}. Any machine learning problem is mathematically formulated as an optimization problem, whose solution determines a set of model parameters (if the \mathcal{M} family is described by a set of parameters – which can be coordinates in a functional database). We can note \mathcal{M}_0 the space of the hyperplanes of \mathbb{R}^p in the sense thatm\in\mathcal{M}_0 \text{\quad means \quad}m(\mathbf{x})=\beta_0+\beta^T\mathbf{x}\text{ where }\beta\in\mathbb{R}^p generating the class of linear predictors. We will then have the estimator that minimizes the empirical risk. Some of the recent work in statistical learning aims to study the properties of the estimator \widehat{m}^\star, known as “oracle”, in a family of \mathcal{M} estimators, \widehat{m}^{\star} =\underset{\widehat{m}\in\mathcal{M}}{\text{argmin}}\big\lbrace\mathcal{R}(\widehat{m},m)\big\rbraceThis estimator is, of course, impossible to define because it depends on m, the real model, unknown.
But let’s come back a little more to these loss functions. A loss function \ell is a function \mathbb{R}^d\times\mathbb{R}^d\rightarrow\mathbb{R}_+, symmetric, which checks the triangular inequality, and such that \ell(x,y)=0 if and only if x=y. The associated norm is \\cdot\, such that \ell(x,y)=\xy\=\ell(xy,0) (using the fact that \ell(x,y+z)=\ell(xy,z) – we will review this fundamental property later).
For a quadratic loss function, it should be noted that we can have a particular interpretation of this problem, since:\overline{y}=\underset{m\in\mathbb{R}}{\text{argmin}} \left\lbrace\sum_{i=1}^n\frac{1}{n} [y_im]^2\right\rbrace=\underset{m\in\mathbb{R}}{\text{argmin}} \left\lbrace \sum_{i=1}^n \ell_2(y_i,m)\right\rbrace where \ell_2 is the usual quadratic distance If we assume – as we did in econometrics – that there is an underlying probabilistic model, and observe that : \displaystyle{\mathbb{E}(Y)=\underset{m\in\mathbb{R}}{\text{argmin}}\left\lbrace\mathbb{E}\left([Ym]^2\right)\right\rbrace=\underset{m\in\mathbb{R}}{\text{argmin}}\left\lbrace\mathbb{E}\big[\ell_2(Y,m)\big]\right\rbrace}it should be noted that what we are trying to obtain here, by solving the problem (6) by taking the norm \ell_2, is an approximation (in a given functional space, \mathcal{M}) of the conditional expectation x\mapsto\mathbb{E}[Y\mathbf{X}=\mathbf{x}]. Another particularly interesting loss function is the loss \ell_1, \ell_1 (y,m)=ym[\latex]. It should be recalled that [latex display="true"]\displaystyle{\text{median}(\boldsymbol{y})=\underset{m\in\mathbb{R}}{\text{argmin}}\left\lbrace\sum_{i=1}^n\ell_1(y_i,m)\right\rbrace}The optimization problem :\widehat{m}^{\star}=\underset{m\in\mathcal{M}_0}{\text{argmin}}\left\lbrace\sum_{i=1}^n\vert y_im(\mathbf{x}_i)\vert\right\rbrace is obtained in econometrics by assuming that the conditional law of Y follows a Laplace law centered on m(\mathbf{x}), and by maximizing the likelihood (log) (the sum of the absolute values of the errors corresponds to the logreasonableness of a Laplace law). It should also be noted that if the conditional law of Y is symmetrical with respect to 0, the median and the mean coincide If this loss function is rewritten \ell_1(y,m)=\vert (ym)(1/2\boldsymbol{1}_{y\leq m})\vert a generalization can be obtained for \tau\in[0.1]:\widehat{m}^\star_\tau=\underset{m\in\mathcal{M}_0}{\text{argmin}}\left\lbrace\sum_{i=1}^n \ell_\tau^{ q} (y_i,m(\mathbf{x}_i)) \right\rbracewhere\ell_{\tau}^{q}(x,y)= (xy)(\tau\boldsymbol{1}_{x\leq y}) is then the quantile regression of level \tau (Koenker, 2003; d'Haultefœuille & Givord, 2014). Another loss function, introduced by Aigner et al (1977) and analysed in Waltrup et al (2014), is the function associated with the notion of expectations: \displaystyle{\ell}^{\text{ e}}_{\tau}(x,y)= (xy)^2\cdot\big\vert\tau\boldsymbol{1}_{x\leq y}\big\vertwith \tau\in[0.1]. We see the parallel with the quantile function: \displaystyle{\ell}^{\text{ q}}_{\tau}(x,y)= \vert xy\vert \cdot\big\vert\tau\boldsymbol{1}_{x\leq y}\big\vertKoenker & Machado (1999) and Yu & Moyeed (2001) also noted a link between this condition and the search for maximum likelihood when Y's conditional law follows an asymmetric Laplace law.
In connection with this approach, Gneiting (2011) introduced the notion of "ellicable statistics"  or "ellicable measurement" in its probabilistic (or distributional) version: a statistic T will be said to be "ellicitable" if there is a loss function \ell:\mathbb{R}\times\mathbb{R}\rightarrow\mathbb{R}_+ such that:T(Y)=\underset{x\in\mathbb{R}}{\text{argmin}}\left\lbrace\int_{\mathbb{R}} \ell(x,y)dF(y)\right\rbrace=\underset{x\in\mathbb{R}}{\text{argmin}}\left\lbrace\mathbb{E}\big[ \ell(x,Y)\big]\text{ where }Y\overset{\mathcal{L}}{\sim} F\right\rbrace The mean (mathematical expectation) is thus ellicable by the quadratic distance, \ell_2, while the median is ellicable by the distance \ell_1. According to Gneiting (2011), this property is essential for obtain predictions and forecasts. There may then be a strong link between measures associated with probabilistic models and loss functions. Finally, Bayesian statistics provide a direct link between the form of the a priori law and the loss function, as studied by Berger (1985) and Bernardo & Smith (2000). We will come back to the use of these different norms in the section on penalization.
To be continued (keep in mind that references are online here)…
[1] Where the indicator \mathbf{1}_{\pm} does not take values 0 or 1 (like the classical \mathbf{1} function), but 1 and +1.
]]>library(XML)
library(stringr)
url="http://www.nserccrsng.gc.ca/NSERCCRSNG/FundingDecisionsDecisionsFinancement/ResearchGrantsSubventionsDeRecherche/ResultsGSCResultatsCSS_eng.asp"
download.file(url,destfile = "GSC.html")
library(XML)
tables=readHTMLTable("GSC.html")
GSC=tables[[1]]$V1
GSC=as.character(GSC[(1:2)])
namesGSC=tables[[1]]$V2
namesGSC=as.character(namesGSC[(1:2)])
Correction = function(x) as.numeric(gsub('[$,]', '', x))
YEAR=2013:2018
for(i in 1:length(YEAR)){
y=YEAR[i]
grants= function(gsc){
url=paste("http://www.nserccrsng.gc.ca/NSERCCRSNG/FundingDecisionsDecisionsFinancement/ResearchGrantsSubventionsDeRecherche/ResultsGSCDetailResultatsCSSDetails_eng.asp?Year=",y,"&GSC=",gsc,sep="")
download.file(url,destfile = "GSC.html")
library(XML)
tables=readHTMLTable("GSC.html")
X=as.character(tables[[1]]$"Awarded Amount")
A=as.numeric(Vectorize(Correction)(X))
return(c(median(A),mean(A),as.numeric(quantile(A,(1:99)/100))))
}
M=Vectorize(grants)(GSC[1:12])
plot(M[3:101,8],(1:99)/100,type="s",xlim=c(0,130000),xlab=
paste("Annual Discovery Grant (CAN)  ",y,sep=""),ylab="")
lines(M[3:101,5],(1:99)/100,type="s",col="red")
lines(M[3:101,4],(1:99)/100,type="s",col="blue")
abline(v=M[3,5],lty=2,col=rgb(1,0,0,.4))
idx=which(M[3:101,8]<M[3,5])
lines(M[2+idx,8],(idx)/100,type="s",lwd=4)
legend("bottomright",c("maths","physics","chemestry"),
col=c("black","red","blue"),lty=1,bty="n")}
With those functions, I plot the cumulative distribution functions for three disciplines, manely maths, physics and chemistry. I added a line for the lowest value in physics (the vertical line), and the bold line shows the proportion of researchers in maths who got less than the lowest amount in physics,
Hence, in 2013, 60% of the researchers in maths get less than any researcher in physics (and more than 90% in maths get less than any researcher in chemistry). Then, from 2014 to 2018, we get
It is rather constant : 50% of the researchers in mathematics in Canada get less than any researcher in physics, or in chemistry. I don’t understand why, but it’s interesting to observe that this is very stable…
]]>
Ahamada, I. & E. Flachaire (2011). NonParametric Econometrics. Oxford University Press.
Aigner, D., Lovell, C.A.J & Schmidt, P. (1977). Formulation and estimation of stochastic frontier production function models. Journal of Econometrics, 6, 21–37.
Aldrich, J. (2010). The Econometricians’ Statisticians, 18951945. History of Political Economy, 42 111–154.
Altman, E., Marco, G. & Varetto, F. (1994). Corporate distress diagnosis: Comparisons using linear discriminant analysis and neural networks (the Italian experience). Journal of Banking & Finance 18, 505–529.
Angrist, J.D. & Lavy, V. (1999). Using Maimonides’ Rule to Estimate the Effect of Class Size on Scholastic Achievement. Quarterly Journal of Economics, 114, 533–575.
Angrist, J.D. & Pischke, J.S. (2010). The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics. Journal of Economic Perspective, 24, 3–30.
Angrist, J.D. & Pischke, J.S. (2015). Mastering Metrics. Princeton University Press.
Angrist, J.D. & Krueger, A.B. (1991). Does Compulsory School Attendance Affect Schooling and Earnings? Quarterly Journal of Economics, 106, 979–1014.
Bottou, L. (2010) LargeScale Machine Learning with Stochastic Gradient Descent Proceedings of the 19th International Conference on Computational Statistics (COMPSTAT’2010), 177–187.
Bajari, P., Nekipelov, D., Ryan, S.P. & Yang, M. 2015. Machine learning methods for demand estimation. American Economic Review, 105 481–485.
Bazen, S. & K. Charni (2015). Do earnings really decline for older workers? AMSE 201511 Discussion Paper, AixMarseille University.
Bellman, R.E. (1957). Dynamic programming. Princeton University Press.
Belloni, A., Chernozhukov, V. & Hansen, C. (2010). Inference Methods for HighDimensional Sparse Econometric Models. Advances in Economics and Econometrics, 245–295
Belloni, A., Chen, D., Chernozhukov, V. & Hansen, C. (2012). Sparse Models and Methods for Optimal Instruments With an Application to Eminent Domain. Econometrica, 80, 2369–2429.
Benjamini, Y. & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57:289–300.
Berger, J.O. (1985). Statistical decision theory and Bayesian Analysis (2nd ed.). SpringerVerlag.
Berk, R.A. (2008). Statistical Learning from a Regression Perspective. Springer Verlag.
Berkson, J. (1944). Applications of the logistic function to bioassay. Journal of the American Statistical Association, 9, 357–365.
Berkson, J. (1951). Why I prefer logits to probits. Biometrics, 7 (4), 327–339.
Bernardo, J.M. & Smith, A.F.M. (2000). Bayesian Theory. John Wiley.
Berndt, E. R. (1990). The Practice of Econometrics: Classic and Contemporary. Addison Wesley.
Bickel, P.J., Gotze, F. & van Zwet, W. (1997). Resampling fewer than observations: gains, losses and remedies for losses. Statistica Sinica, 7, 131.
Bishop, C. (2006). Pattern Recognition and Machine Learning. Springer Verlag.
Blanco, A. PinoMejias, M., Lara, J. & Rayo, S. (2013). Credit scoring models for the microfinance industry using neural networks: Evidence from peru. Expert Systems with Applications, 40, 356–364.
Bliss, C.I. (1934). The method of probits. Science, 79, 38–39.
Blumer, A., Ehrenfeucht, A., Haussler, D. & Warmuth, M.K. (1989). Learnability and the VapnikChervonenkis dimension. Journal of the ACM, 36:4, 929–965.
Breiman, L. Fiedman, J., Olshen, R.A. & Stone, C.J. (1984). Classification And Regression Trees. Chapman & Hall/CRC.
Breiman, L. (1995). Better Subset Regression Using the Nonnegative Garrote. Technometrics, 37:4, 373–384.
Breiman, L. (2001a). Statistical Modeling: The Two Cultures. Statistical Science, 16:3, 199–231.
Breiman, L. (2001b). Random forests. Machine learning, 45:1, 5–32.
Brown, L.D. (1986) Fundamentals of statistical exponential families: with applications in statistical decision theory. Institute of Mathematical Statistics, Hayworth, CA, USA.
Bühlmann, P. & van de Geer, S. (2011). Statistics for High Dimensional Data: Methods, Theory and Applications. Springer Verlag.
Candès, E. & Plan, Y. (2009). Nearideal model selection by minimization. The Annals of Statistics, 37:5, 2145–2177.
Clarke, B.S., Fokoué, E. & Zhang, H.H. (2009). Principles and Theory for Data Mining and Machine Learning. Springer Verlag.
Cortes, C. & Vapnik, V. (1995). Supportvector networks. Machine Learning 20 273–297.
Cover, T.M. (1965). Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition. IEEE Transactions on Electronic Computers, 14:3, 326–334.
Cover, T.M. & Hart, P. (1965). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13:1, 21 – 27.
Cover, T.M. & Thomas, J. (1991). Elements of Information Theory. Wiley.
Cybenko, G. (1989). Approximation by Superpositions of a Sigmoidal Function. Mathematics of Control, Signals, and Systems, 2, 303–314.
Darmois, G. (1935). Sur les lois de probabilites a estimation exhaustive. Comptes Rendus de l’Académie des Sciences, Paris, 200 1265–1266.
Daubechies, I., Defrise, M. & De Mol, C. (2004). An iterative thresholding algorithm for linear inverse problems with sparsity constraint. Communications on Pure and Applied Mathematics, 57:11, 1413–1457
Davison, A.C. (1997). Bootstrap. Cambridge University Press.
Davidson, R. & MacKinnon, J.G. (1993). Estimation and Inference in Econometrics. Oxford University Press.
Davidson, R. & MacKinnon, J.G. (2003). Econometric Theory and Methods. Oxford University Press.
Duo, Q. (1993). The Formation of Econometrics. Oxford University Press.
Debreu, G. (1986). Theoretic Models: Mathematical Form and Economic Content. Econometrica, 54, 1259–1270.
Dhillon, P., Lu, Y. Foster, D.P. & Ungar, L.H. (2014). New Subsampling Algorithms for Fast Least Squares Regression. in Advances in Neural Information Processing Systems 26, Burges, Bottou, Welling, Ghahramani & Weinberger Eds., Curran Associates.
Efron, B. & Tibshirani, R. (1993). Bootstrap. Chapman Hall CRC.
Engel, E. (1857). Die Productions und Consumtionsverhältnisse des Königreichs Sachsen. Statistisches Bureau des Königlich Sächsischen Ministeriums des Innern.
Feldstein, M. & Horioka, C. (1980). Domestic Saving and International Capital Flows. Economic Journal, 90, 314–329.
Flach, P. (2012). Machine Learning. Cambridge University Press.
Foster, D.P. & George, E.I. (1994). The Risk Inflation Criterion for Multiple Regression. The Annals of Statistics, 22:4, 1947–1975.
Friedman, J.H. (1997). Data Mining and Statistics: What’s the Connection. Proceedings of the 29th Symposium on the Interface Between Computer Science and Statistics.
Frisch, R. & Waugh, F.V. (1933). Partial Time Regressions as Compared with Individual Trends. Econometrica. 1, 387–401.
Gneiting, T. (2011). Making and Evaluating Point Forecasts. Journal of the American Statistical Association, 106, 746–762.
Givord, P. (2010). Méthodes économétriques pour l’évaluation de politiques publiques. INSEE Document de Travail, 08
Grandvalet, Y., Mariéthoz, J., & Bengio, S. 2005. A probabilistic interpretation of SVMs with an application to unbalanced classification. Advances in Neural Information Processing Systems 18.
Groves, T. & Rothenberg, T. (1969). A note on the expected value of an inverse matrix. Biometrika, 56:3, 690–691.
Haavelmo, T. (1944). The probability approach in econometrics, Econometrica, 12:iiivi and 1–115.
Hastie, T. & Tibshirani, R. (1990). Generalized Additive Models. Chapman & Hall/CRC.
Hastie, T., Tibshirani, R. & Friedman, J. (2009). The Elements of Statistical Learning. Springer Verlag.
Hastie, T., Tibshirani, W. & Wainwright, M. (2015). Statistical Learning with Sparsity. Chapman CRC.
Hastie, T., Tibshiriani, R. & Tibshiriani, R.J. (2016). Extended comparisons of best subset selection, forward stepwise selection and the Lasso. ArXiV, 1707.08692.
d’Haultefœuille, X. & Givord, P. (2014) La régression quantile en pratique. économie & Statistiques, 471, 85–111.
Hebb, D.O. (1949). The organization of behavior, New York, Wiley.
Heckman, J.J. (1979). Sample selection bias as a specification error. Econometrica, 47, 153–161.
Heckman, J.J., Tobias, J.L. & Vytlacil, E. (2003). Simple Estimators for Treatment Parameters in a LatentVariable Framework. The Review of Economics and Statistics, 85, 748–755.
Hendry, D F. & Krolzig, H.M. (2001). Automatic Econometric Model Selection. Timberlake Press.
Herbrich, R., Keilbach, M., Graepel, T. BollmannSdorra, P. & Obermayer, K. (1999). Neural Networks in Economics. in Computational Techniques for Modelling in Economics, T. Brenner Eds. Springer Verlag, 169–196.
Hoerl, A.E. (1962). Applications of ridge analysis to regression problems. Chemical Engineering Progress, 58:3, 54–59.
Hoerl, A.E. & Kennard, R.W. (1981). Ridge regression: biased estimation for nonorthogonal problems This Week’s Citation Classic, ISI.
Holland, P. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–960.
Hyndman, R. , Koehler, A.B., Ord, J.K. & Snyder, R.D. (2009). Forecasting with Exponential Smoothing. Springer Verlag.
James, G., D. Witten, T. Hastie, & R. Tibshirani (2013). An introduction to Statistical Learning. Springer Series in Statistics.
Khashman, A. (2011). Credit risk evaluation using neural networks: Emotional versus conventional models. Applied Soft Computing, 11, 5477–5484.
Kean, M.P. (2010). Structural vs. atheoretic approaches to econometrics. Journal of Econometrics, 156, 3–20.
Kleiner, A., Talwalkar, A., Sarkar , P. & Jordan, M. (2012). The Big Data Bootstrap. arXiv:1206.6415 .
Koch, I. (2013). Analysis of Multivariate and HighDimensional Data. Cambridge University Press.
Koenker, R. (1998). Galton, Edgeworth, Frish, and prospects for quantile regression in Econometrics. Conference on Principles of Econometrics, Madison.
Koenker, R. (2003). Quantile Regression. Cambridge University Press.
Koenker, R. & Machado, J. (1999). Goodness of fit and related inference processes for quantile regression Journal of the American Statistical Association, 94, 12961309.
Kolda, T. G. & Bader, B. W. (2009). Tensor decompositions and applications. SIAM Review 51, 455–500.
Koopmans, T.C. (1957). Three Essays on the State of Economic Science. McGrawHill.
Kuhn, M. & Johnson, K. (2013). Applied Predictive Modeling. Springer Verlag.
Landis, J.R. & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.
LeCun, Y., Bengio, Y. & Hinton, G. (2015). Deep learning. Nature 521 436–444.
Leeb, H. (2008). Evaluation and selection of models for outofsample prediction when the sample size is small relative to the complexity of the datagenerating process. Bernoulli 14:3, 661–690.
Lemieux, T. (2006). The « Mincer Equation » Thirty Years After Schooling, Experience, and Earnings. in Jacob Mincer A Pioneer of Modern Labor Economics, Grossbard Eds, 127–145, Springer Verlag.
Li, J. & J. S. Racine (2006). Nonparametric Econometrics. Princeton University Press.
Li, C., Li, Q., Racine, J. & Zhang, D. (2017). Optimal Model Averaging Of Varying Coefficient Models. Department of Economics Working Papers 201701, McMaster University.
Lin, H.W., Tegmark, M. & Rolnick, D. (2016). Why does deep and cheap learning work so well? ArXiv 1608.08225.
Lucas, R.E. (1976). Econometric Policy Evaluation: A Critique. CarnegieRochester Conference Series on Public Policy, 19–46.
Mallows, C.L. (1973). Some Comments on Cp . Technometrics, 15, 661–675.
McCullogh, W.S. & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5:4, 115–133.
Mincer, J. (1974). Schooling, experience and earnings. Columbia University Press.
Mitchell, T. (1997). Machine Learning. McGrawHill.
Morgan, J.N. & Sonquist, J.A. (1963). Problems in the analysis of survey data, and a proposal. Journal of the American Statistcal Association, 58, 415–434.
Morgan, M.S. (1990). The history of econometric ideas. Cambridge University Press.
Mohri, M., Rostamizadeh, A. & Talwalker, A. (2012) Foundations of Machine Learning. MIT Press.
Mullainathan, S. & Spiess, J. (2017). Machine learning: An applied econometric approach. Journal of Economic Perspectives, 31 87–106.
Müller, M. (2011). Generalized Linear Models in Handbook of Computational Statistics, J.E Gentle, W.K. Härdle & Y. Mori Eds. Springer Verlag.
Murphy, K.R. (2012). Machine Learning: a Probabilistic Perspective. MIT Press.
Murphy, K. M. & F. Welch (1990). Empirical ageearnings profiles. Journal of Labor Economics 8, 202–229.
Nadaraya, E. A. (1964). On Estimating Regression. Theory of Probability and its Applications, 9:1, 141–2.
Natarajan, B. K. (1995). Sparse approximate solutions to linear systems. SIAM Journal on Computing (SICOMP), 24 227–234.
Nevo, A. & Whinston, M.D. (2010). Taking the Dogma out of Econometrics: Structural Modeling and Credible Inference. Journal of Economic Perspective, 24, 69–82.
Neyman, J. (1923). Sur les applications de la théorie des probabilités aux expériences agricoles : Essai des principes. Mémoire de master, republibé dans Statistical Science, 5, 463–472.
Nisbet, R., Elder, J. & Miner, G. (2011). Handbook of Statistical Analysis and Data Mining Applications. Academic Press, New York.
Okun, A. (1962). Potential GNP: Its measurement and significance. Proceedings of the Business and Economics Section of the American Statistical Association, 98–103.
Orcutt, G.H. (1952). Toward a partial redirection of econometrics. Review of Economics and Statistics, 34 195–213.
Pagan, A. & A. Ullah (1999). Nonparametric Econometrics. Themes in Modern Econometrics. Cambridge: Cambridge University Press.
Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. Philosophical Magazine, 2, 559–572.
Platt, J. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers. 10, 61–74.
Portnoy, S. (1988). Asymptotic Behavior of Likelihood Methods for Exponential Families when the Number of Parameters Tends to Infinity. Annals of Statistics, 16:356–366.
Quenouille, M. H. (1949). Problems in Plane Sampling. The Annals of Mathematical Statistics 20(3):355–375.
Quenouille, M. H. (1956). Notes on Bias in Estimation. Biometrika 43(34), 353–360.
Quinlan, J.R. (1986). Induction of decision trees. Machine Learning 1 81–106.
Reiersøl, O. (1945). Confluence analysis of means of instrumental sets of variables. Arkiv. for Mathematik, Astronomi Och Fysik, 32.
Rosenbaum, P. & Rubin, D. (1983). The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika, 70, 41–55.
Rosenblatt, F. (1958). The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review, 65, 386–408.
Rubin, D. (1974). Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies. Journal of Educational Psychology, 66, 688–701.
Ruppert, D., Wand, M. P. & Carroll, R.J. (2003). Semiparametric Regression. Cambridge University Press.
Samuel, A. (1959). Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development, 44:1.
Schultz, H. (1930). The Meaning of Statistical Demand Curves. University of Chicago.
Shao, J. (1993). Linear Model Selection by CrossValidation. Journal of the American Statistical Association 88:(422), 486–494.
ShalevShwartz, S. & BenDavid, S. (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press.
Shao, J. (1997). An Asymptotic Theory for Linear Model Selection. Statistica Sinica, 7, 221–264.
Shapire, R.E. & Freund, Y. (2012). Boosting. MIT Press.
Silverman, B.W. (1986) Density Estimation. Chapman & Hall. Simonoff, J. S. (1996). Smoothing Methods in Statistics. Springer.
Stone, M. (1977). An Asymptotic Equivalence of Choice of Model by CrossValidation and Akaike’s Criterion. Journal of the Royal Statistical Society. Series B , 39:1, 44–47.
Tam, K.Y. & Kiang, M.Y. (1992). Managerial applications of neural networks: The case of bank failure predictions. Management Science, 38, 926–947.
Tan, H. (1995). NeuralNetwork model for stock forecasting. MSc Thesis, Texas Tech. University.
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B., 58, 267–288.
Tibshirani, R. & Wasserman, L. (2016). A Closer Look at Sparse Regression.
Tikhonov, A. N. (1963). Solution of incorrectly formulated problems and the regularization method. Soviet Mathematics, 4: 1035–1038.
Tinbergen, J. (1939). Statistical Testing of Business Cycle Theories. Vol. 1: A Method and its Application to Investment activity; Vol. 2: Business Cycles in the United States of America, 1919—1932. Geneva: League of Nations.
Tobin, J. (1958). Estimation of Relationship for Limited Dependent Variables. Econometrica, 26, 24–36.
Tropp, (2011). Improved analysis of the subsampled randomized Hadamard transform. Advances in Adaptive Data Analysis, 3:1, 115–126.
Tsen, P. (2001). Convergence of a block coordinate descent for nondifferentiable minization. Journal of Optimization Theory and Applications, 109:3, 475–494.
Tufféry, S. (2001). Data Mining and Statistics for Decision Making. Wiley Interscience.
Tukey, J. W. (1958). Bias and confidence in not quite large samples. The Annals of Mathematical Statistics, 29:614–623.
Vailiant, L.G. (1984). A theory of the learnable. Communications of the ACM 27:11, 1134–1142.
Vapnik, V. (1998). Statistical Learning Theory. Wiley.
Vapnik, C., & Chervonenkis, A. (1971). On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications, 16:264–280.
Varian, H.R. (2014). Big Data: New Tricks for Econometrics. Journal of Economic Perspectives, 28(2):3–28.
Vert, J.P. (2017). Machine learning in computational biology. ENSAE.
Waltrup, L.S., Sobotka, F., Kneib, T. & Kauermann, G. (2014). Expectile and quantile regression—David and Goliath? Statistical Modelling, 15, 433 – 456.
Watson, G. S. (1964). Smooth regression analysis. Sankhya: The Indian Journal of Statistics, Series A, 26:4, 359–372.
Watt, J., Borhani, R. & Katsaggelos, A. (2016). Machine Learning Refined : Foundations, Algorithms, and Applications. Cambridge University Press.
White, H. (1989). Learning in Artificial Neural Networks: A Statistical Perspective. Neural Computation, 1:4, 425–464.
Widrow, B. & Hoff, M.E. Jr. (1960). Adaptive Switching Circuits. IRE WESCON Convention Record, 4:96–104.
Wolpert, D.H., Macready, W.G. (1997), No Free Lunch Theorems for Optimization, IEEE Transactions on Evolutionary Computation 1, 67.
Wolpert, David (1996), The Lack of A Priori Distinctions between Learning Algorithms, Neural Computation, 13411390.
Working, E. J. (1927). What do statistical ‘demand curves’ show? Quarterly Journal of Economics, 41:212–35.
Yu, K. & Moyeed, R. (2001). Bayesian quantile regression. Statistics & Probability Letters, 54, 437–447.
Zinkevich M.A., Weimer, M., Smola, A. & Li, L. (2010). Parallelized Stochastic Gradient. Advances in neural information processing systems, 2595–2603.
]]>In parallel with these tools developed by, and for economists, a whole literature has been developed on similar issues, centered on the problems of prediction and forecasting. For Breiman (2001a), a first difference comes from the fact that the statistic has developed around the principle of inference (or to explain the relationship linking y to variables \mathbf{x}) while another culture is primarily interested in prediction. In a discussion that follows the article, David Cox states very clearly that in statistic (and econometrics) “predictive success (…) is not the primary basis for model choice“. We will get back here on the roots of automatic learning techniques. The important point, as we will see, is that the main concern of machine learning is related to the generalization properties of a model, i.e. its performance – according to a criterion chosen a priori – on new data, and therefore on nonsample tests.
Today, we speak of “machine learning” to describe a whole set of techniques, often computational, as alternatives to the classical econometric approach. Before characterizing them as much as possible, it should be noted that historically other names have been given. For example, Friedman (1997) proposes to make the link between statistics (which closely resemble econometric techniques – hypothesis testing, ANOVA, linear regression, logistics, GLM, etc.) and what was then called “data mining” (which then included decision trees, methods from the closest neighbours, neural networks, etc.). The bridge between those two cultures corresponds to “statistical learning” techniques described in Hastie et al (2009). But one should keep in mind that machine learning is a very large field of research.
The socalled “natural” learning (as opposed to machine learning) is that of children, who learn to speak, read and play. Learning to speak means segmenting and categorizing sounds, and associating them with meanings. A child also learns simultaneously the structure of his or her mother tongue and acquires a set of words describing the world around him or her. Several techniques are possible, ranging from rote learning, generalization, discovery, more or less supervised or autonomous learning, etc. The idea in artificial intelligence is to take inspiration from the functioning of the brain to learn, to allow “artificial” or “automatic” learning, by a machine. A first application was to teach a machine to play a game (tictactoe, chess, go, etc.). An essential step is to explain the objective it must achieve to win. One historical approach has been to teach the machine the rules of the game. If it allows you to play, it will not help the machine to play well. Assuming that the machine knows the rules of the game, and that it has a choice between several dozen possible moves, which one should it choose? The classical approach in artificial intelligence uses the socalled minmax algorithm using an evaluation function: in this algorithm, the machine searches forward in the possible moves tree, as far as the calculation resources allow (about ten moves in chess, for example). Then, it calculates different criteria (which have been previously indicated to her) for all positions (number of pieces taken, or lost, occupancy of the center, etc. in our example of the chess game), and finally, the machine plays the move that allows it to maximize its gain. Another example may be the classification and recognition of images or shapes. For example, the machine must identify a number in a handwritten handwriting (checks, ZIP code on envelopes, etc). It is a question of predicting the value of a variable y, knowing that a priori y\in\{0,1,2,\cdots,8,9\}. A classical strategy is to provide the machine with learning bases, in other words here millions of labelled (identified) images of handwritten numbers. A simple (and natural) strategy is to use a decision criterion based on the closest neighbors whose labels are known (using a predefined metric).
The method of the closest neighbors (“knearest neighbors”) can be described as follows: we consider (as in the previous part) a set of n observations, i. e. pairs (y_i,\mathbf{x}_i) with \mathbf{x}_i\in\mathbb{R}^p. Let us consider a distance \Delta on \mathbb{R}^p (the Euclidean distance or the Mahalanobis distance, for example). Given a new observation \mathbf{x}\in\mathbb{R}^p, let us assume the ordered observations as a function of the distance between the \mathbf{x}_i and \mathbf{x}, in the sense that \Delta(\mathbf{x}_1, \mathbf{x})\leq\Delta(\mathbf{x}_2, \mathbf{x})\leq\cdots\leq\Delta(\mathbf{x}_n, \mathbf{x}) then we can consider as prediction for y the average of the nearest k neighbours,\widehat{m}_k(\mathbf{x})=\frac{1}{k}\sum_{i=1}^k y_iLearning here works by induction, based on a sample (called the learning – or training – sample).
Automatic learning includes those algorithms that give computers the ability to learn without being explicitly programmed (as Arthur Samuel defined it in 1959). The machine will then explore the data with a specific objective (such as searching for the nearest neighbours in the example just described). Tom Mitchell proposed a more precise definition in 1998: a computer program is said to learn from experience E in relation to a task T and a performance measure P, if its performance on T, measured by P, improves with experience E. Task T can be a defect score for example, and performance P can be the percentage of errors made. The system learns if the percentage of predicted defects increases with experience.
As we can see, machine learning is basically a problem of optimizing a criterion based on data (from now on called learning). Many textbooks on machine learning techniques propose algorithms, without ever mentioning any probabilistic model. In Watt et al (2016) for example, the word “probability” is mentioned only once, with this footnote that will surprise and make smile any econometricians, “the logistic regression can also be interpreted from a probabilistic perspective” (page 86). But many recent books offer a review of machine learning approaches using probabilistic theories, following the work of Vaillant and Vapnik. By proposing the paradigm of “probably almost correct” learning (PAC), a probabilistic flavor has been added to the previously very computational approach, by quantifying the error of the learning algorithm (usually in a classification problem).
To be continued (references are online here)…
In the Gaussian linear model, the determination coefficient – noted R^2 – is often used as a measure of fit quality. It is based on the variance decomposition formula \underbrace{\frac{1}{n}\sum_{i=1}^n (y_i\bar{y})^2}_{\text{total variance}}=\underbrace{\frac{1}{n}\sum_{i=1}^n (y_i\widehat{y}_i)^2}_{\text{residual variance}}+\underbrace{\frac{1}{n}\sum_{i=1}^n (\widehat{y}_i\bar{y})^2}_{\text{explained variance}} The R^2 is defined as the ratio of explained variance and total variance, another interpretation of the coefficient that we had introduced from the geometry of the least squares R^2= \frac{\sum_{i=1}^n (y_i\bar{y})^2\sum_{i=1}^n (y_i\widehat{y}_i)^2}{\sum_{i=1}^n (y_i\bar{y})^2}The sums of the error squares in this writing can be rewritten as a loglikelihood. However, it should be remembered that, up to one additive constant (obtained with a saturated model) in generalized linear models, deviance is defined by {Deviance}(\widehat{\beta}) = 2\log[\mathcal{L}] which can also be noted Deviance(\widehat{\mathbf{y}}). A null deviance can be defined as the one obtained without using the explanatory variables \mathbf{x}, so that \widehat{y}_i=\overline{y}. It is then possible to define, in a more general context (with a nonGaussian distribution for y)R^2=\frac{{Deviance}(\overline{y}){Deviance}(\widehat{\mathbf{y}})}{{Deviance}(\overline{y})}=1\frac{{Deviance}(\widehat{\mathbf{y}})}{{Deviance}(\overline{y})}However, this measure cannot be used to choose a model, if one wishes to have a relatively simple model in the end, because it increases artificially with the addition of explanatory variables without significant effect. We will then tend to prefer the adjusted R^2,\bar R^2 = {1(1R^{2})\cdot{n1 \over np}} = R^{2}\underbrace{(1R^{2})\cdot{p1 \over np}}_{\text{penalty}}where p is the number of parameters of the model. Measuring the quality of fit will penalize overly complex models.
This idea will be found in the Akaike criterion, where AIC=Deviance+2\cdot p or in the Schwarz criterion, BIC=Deviance+log(n)\cdot p. In large dimensions (typically p>\sqrt{n}), we will tend to use a corrected AIC, defined by AIC_c=Deviance+2⋅p⋅n/(np1) .
These criterias are used in socalled “stepwise” methods, introducing the set methods. In the “forward” method, we start by regressing to the constant, then we add one variable at a time, retaining the one that lowers the AIC criterion the most, until adding a variable increases the AIC criterion of the model. In the “backward” method, we start by regressing on all variables, then we remove one variable at a time, removing the one that lowers the AIC criterion the most, until removing a variable increases the AIC criterion from the model.
Another justification for this notion of penalty (we will come back to this idea in machine learning) can be the following. Let us consider an estimator in the class of linear predictors, \mathcal{M}=\big\lbrace m:~m(\mathbf{x})=s_h(\mathbf{x})^T\mathbf{y} \text{ where }S=(s(\mathbf{x}_1),\cdots,s(\mathbf{x}_n))^T\text{ is some smoothing matrix}\big\rbrace and assume that y=m_0 (x)+\varepsilon, with \mathbb{E}[\varepsilon]=0 and Var[\varepsilon]=\sigma^2\mathbb{I}, so that m_0 (x)=\mathbb{E}[YX=x] . From a theoretical point of view, the quadratic risk, associated with an estimated model \widehat{m}, \mathbb{E}\big[(Y\widehat{m}(\mathbf{X}))^2\big], is written\mathcal{R}(\widehat{m})=\underbrace{\mathbb{E}\big[(Ym_0(\mathbf{X}))^2\big]}_{\text{error}}+\underbrace{\mathbb{E}\big[(m_0(\mathbf {X})\mathbb{E}[\widehat{m}(\mathbf{X})])^2\big]}_{\text{bias}^2}+\underbrace{\mathbb{E}\big[(\mathbb{E}[\widehat{m}(\mathbf{X})]\widehat{m}(\mathbf{X}))^2\big]}_{\text{variance}} if m_0 is the true model. The first term is sometimes called “Bayes error”, and does not depend on the estimator selected, \widehat{m}.
The empirical quadratic risk, associated with a model m, is here: \widehat{\mathcal{R}}_n(m)=\frac{1}{n}\sum_{i=1}^n (y_im(\mathbf{x}_i))^2 (by convention). We recognize here the mean square error, “mse”, which will more generally give the “risk” of the model m when using another loss function (as we will discuss later on). It should be noted that:\displaystyle{\mathbb{E}[\widehat{\mathcal{R}}_n(m)]=\frac{1}{n}\m_0(\mathbf{x})m(\mathbf{x})\^2+\frac{1}{n}\mathbb{E}\big(\{Y}m_0(\mathbf{X})\^2\big)} We can show that:n\mathbb{E}\big[\widehat{\mathcal{R}}_n(\widehat{m})\big]=\mathbb{E}\big(\Y\widehat{m}(\mathbf{x})\^2\big)=\(\mathbb{I}\mathbf{S})m_0\^2+\sigma^2\\mathbb{I}\mathbf{S}\^2so that the (real) risk of \widehat{m} is: {\mathcal{R}}_n(\widehat{m})=\mathbb{E}\big[\widehat{\mathcal{R}}_n(\widehat{m})\big]+2\frac{\sigma^2}{n}\text{trace}(\boldsymbol{S})So, if \text{trace}(\boldsymbol{S})\geq0 (which is not a too strong assumption), the empirical risk underestimates the true risk of the estimator. Actually, we recognize here the number of degrees of freedom of the model, the righthand term corresponding to Mallow’s C_p, introduced in Mallows (1973) using not deviance but R^2.
The most traditional test in econometrics is probably the significance test, corresponding to the nullity of a coefficient in a linear regression model. Formally, it is the test of H_0:\beta_k=0 against H_1:\beta_k\neq 0. The socalled Student test, based on the statistics t_k=\widehat{\beta}_k/se_{\widehat{β}_k}, allows to decide between the two alternatives, using the test pvalue, defined by \mathbb{P}[T>t_k] avec T\overset{\mathcal{L}}{\sim} Std_\nu, where \nu is the number of degrees of freedom of the model (\nu=p+1 for the standard linear model). In large dimension, however, this statistic is of very limited interest, given a significant FDR (“False Discovery Ratio”). Classically, with a level of significance \alpha=0.05, 5% of the variables are falsely significant. Suppose that we have p=100 explanatory variables, but that 5 (only) are really significant. We can hope that these 5 variables will pass the Student test, but we can also expect that 5 additional variables (false positive test) will emerge. We will then have 10 variables perceived as significant, while only half are significant, i.e. an FDR ratio of 50%. In order to avoid this recurrent pitfall in multiple tests, it is natural to use the procedure of Benjamini & Hochberg (1995).
Econometric models are used to implement public policy evaluations. It is therefore essential to fully understand the underlying mechanisms in order to know which variables actually make it possible to act on a variable of interest. But then we move on to another important dimension of econometrics. Jerry Neyman was responsible for the first work on the identification of causal mechanisms, and then Rubin (1974) formalized the test, called the “Rubin causal model” in Holland (1986). The first approaches to the notion of causality in econometrics were based on the use of instrumental variables, models with discontinuity of regression, analysis of differences in differences, and natural or unnatural experiments. Causality is usually inferred by comparing the effect of a policy – or more generally of a treatment – with its counterfactual, ideally given by a random control group. The causal effect of the treatment is then defined as \Delta=y_1y_0, i.e. the difference between what the situation would be with treatment (noted t=1) and without treatment (noted t=0). The concern is that only y=t\cdot y_1+(1t)\cdot y_0 and t are observed. In other words, the causal effect of variable t on t is not observed (since only one of the two potential variables – y_0 or y_1 is observed for each individual), but it is also individual, and therefore a function of xcovariates. Generally, by making assumptions about the distribution of the triplet (Y_0,Y_1,T) , some parameters of the causal effect distribution become identifiable, based on the density of the observable variables (Y,T) . Classically, we will be interested in the moments of this distribution, in particular the average effect of treatment in the population, \mathbb{E}[\Delta] , or even just the average effect of treatment in the case of treatment \mathbb{E}[\DeltaT=1] . If the result (Y_0,Y_1) is independent of the processing access variable T, it can be shown that \mathbb{E}[\Delta]=\mathbb{E}[YT=1] \mathbb{E} [YT=0]. But if this independence hypothesis is not verified, there is a selection bias, often associated with \mathbb{E}[Y_0T=1] \mathbb{E} [Y_0T=0]. Rosenbaum & Rubin (1983) propose to use a propensity to be treated score, p(x)=\mathbb{P}[T=1X=x] , noting that if variable Y_0\ is independent of access to treatment T conditionally to the explanatory variables X, then it is independent of T conditionally to the score p(X) : it is sufficient to match them using their propensity score. Heckman et al (2003) thus proposes a kernel estimator on the propensity score, which simply provides an estimator of the effect of the treatment, provided that it is treated.
To be continued… next time, we’ll introduce “machine learning techniques” (references mentioned above are online here)
The Gaussian linear model is a special case of a large family of linear models, obtained when the conditional distribution of Y (given the covariates) belongs to the exponential family f(y_i\theta_i,\phi)=\exp\left(\frac{y_i\theta_ib(\theta_i)}{a(\phi)}+c(y_i,\phi)\right) with \theta_i=\psi(\mathbf{x}_i^T \beta). Functions a, b and c are specified according to the type of exponential law (studied extensively in statistics since Darmoix (1935), as Brown (1986) reminds us), and \psi is a onetoone mapping that the user must specify. Loglikelihood then has a simple expression \log\mathcal{L}(\mathbf{\theta},\phi\mathbf{y}) =\frac{\sum_{i=1}^ny_i\theta_i\sum_{i=1}^nb(\theta_i)}{a(\phi)}+\sum_{i=1}^n c(y_i,\phi) and the first order condition is then written \frac{\partial \log \mathcal{L}(\mathbf{\theta},\phi\mathbf{y})}{\partial \mathbf{\beta}} = \mathbf{X}^T\mathbf{W}^{1}[\mathbf{y}\widehat{\mathbf{y}}]=\mathbf{0} based on Müller’s (2011) notations, where \mathbf{W} is a weight matrix (which depends on \beta). Given the link between \theta and the expectation of Y, instead of specifying the function \psi(\cdot) , we will tend to specify the link function g(\cdot) defined by \widehat{y}=m(\mathbf{x})=\mathbb{E}[Y\mathbf{X}=\mathbf{x}]=g^{1} (\mathbf{x}^T \beta) For the Gaussian linear regression we consider an identity link, while for the Poisson regression, the natural link (called canonical) is the logarithmic link. Here, as \mathbf{W} depends on \beta (with \mathbf{W}=diag(\nabla g(\widehat{\mathbf{y}})Var[\mathbf{y}]) there is generally no explicit formula for the maximum likelihood estimator. But an iterative algorithm makes it possible to obtain a numerical approximation. By setting \mathbf{z}=g(\widehat{\mathbf{y}})+(\mathbf{y}\widehat{\mathbf{y}})\cdot\nabla g(\widehat{\mathbf{y}}) corresponding to the error term of a Taylor development in order 1 of g, we obtain an algorithm of the form\widehat{\beta}_{k+1}=[\mathbf{X}^T \mathbf{W}_k^{1} \mathbf{X}]^{1} \mathbf{X}^T \mathbf{W}_k^{1} \mathbf{z}_kBy iterating, we will define \widehat{\beta}=\widehat{\beta}_{\infty}, and we can show that – with some additional technical assumptions (detailed in Müller (2011)) – this estimator is asymptotically Gaussian, with \sqrt{n}(\widehat{\beta} \beta)\overset{\mathcal{L}}{\rightarrow} \mathcal{N}(\mathbf{0},I(β)^{1}) where numerically I(\beta)=\varphi\cdot[\mathbf{X}^T \mathbf{W}_\infty^{1} \mathbf{X}] .
From a numerical point of view, the computer will solve the firstorder condition, and actually, the law of Y does not really intervene. For example, one can estimate a “Poisson regression” even when observations are not integers (but they need to be positive). In other words, the law of Y is only an interpretation here, and the algorithm could be introduced in a different way (as we will see later on), without necessarily having an underlying probabilistic model.
Logistic regression is the generalized linear model obtained with a Bernoulli’s law, and a link function which is the quantile function of a logistic law (which corresponds to the canonical link in the sense of the exponential family). Taking into account the form of Bernoulli’s law, econometrics proposes a model for y_i\in\{0,1\}, in which the logarithm of the odds follows a linear model: \log\left(\frac{\mathbb{P}[Y=1\vert \mathbf{X}=\mathbf{x}]}{\mathbb{P}[Y\neq 1\vert \mathbf{X}=\mathbf{x}]}\right)=\beta_0+\mathbf{x}^T\beta or \mathbb{E}[Y\mathbf{X}=\mathbf{x}]=\mathbb{P}[Y=1\mathbf{X}=\mathbf{x}]=\frac{e^{\beta_0+\mathbf{x}^T\beta}}{1+ e^{\beta_0+\mathbf{x}^T\beta}}=H(\beta_0+\mathbf{x}^T\beta) where H(\cdot)=\exp(\cdot)/(1+exp(\cdot)) is the cumulative distribution function of the logistic law. The estimation of (\beta_0,\beta) is performed by maximizing the likelihood: \mathcal{L}=\prod_{i=1}^n \left(\frac{e^{\mathbf{x}_i^T\mathbf{\beta}}}{1+e^{\boldsymbol{x}_i^T\mathbf{\beta}}}\right)^{y_i}\left(\frac{1}{1+e^{\mathbf{x}_i^T\mathbf{\beta}}}\right)^{1y_i} It is said to be a linear models because isoprobability curves here are the parallel hyperplanes b+\mathbf{x}^T\beta . Rather than this model, popularized by Berkson (1944), some will prefer the probit model (see Berkson, 1951), introduced by Bliss (1934). In this model: \mathbb{E}[Y\mathbf{X}=\mathbf{x}]=\mathbb{P}[Y=1\mathbf{X}=\mathbf{x}]=\Phi (\beta_0+\mathbf{x}^T\beta)
where \Phi denotes the distribution function of the reduced centred normal distribution. This model has the advantage of having a direct link with the Gaussian linear model, since y_i=\mathbf{1}(y_i^\star>0) with y_i^\star=\beta_0+\mathbf{x}^T \beta+\varepsilon_i where the residuals are Gaussian, \mathcal{N}(0,\sigma^2). An alternative is to have centered residuals of unit variance, and to consider a latent modeling of the form y_i=\mathbf{1}(y_i^\star>\xi) (where \xi will be fixed). As we can see, these techniques are fundamentally linked to an underlying stochastic model. In the body of the article, we present several alternative techniques – from the learning literature – for this classification problem (with two classes, here 0 and 1).
As we mentioned earlier, the first order condition \mathbf{X}^T (\mathbf{X}\widehat{\beta}\mathbf{y})=\mathbf{0} is solved numerically by performing a QR decomposition, at a cost which consists in O(np^2) operations (where p is the rank of \mathbf{X}^T \mathbf{X}). Numerically, this calculation can be long (either because p is large or because n is large), and a simpler strategy may be to subsample. Let n_s\ll n, and consider a subsample size n_s of \{1,\cdots,n\}. Then \widehat{\beta}_s=(\mathbf{X}_s^T \mathbf{X}_s )^{1} \mathbf{X}_s^T\mathbf{y}_s is a good approximation of \beta as shown by Dhillon et al. (2014). However, this algorithm is dangerous if some points have a high leverage (i.e. L_i=\mathbf{x}_i(\mathbf{X}^T\mathbf{X})^{1}\mathbf{x}_i^T). Tropp (2011) proposes to transform the data (in a linear way), but a more popular approach is to do nonuniform subsampling, with a probability related to the influence of observations (defined by I_i=\widehat{\varepsilon}_iL_i/(1L_i)^2 , and which unfortunately can only be calculated once the model is estimated).
In general, we will talk about massive data when the data table of size does not fit in the RAM memory of the computer. This situation is often encountered in statistical learning nowadays with very often p\ll n. This is why, in practice, many libraries of algorithms assimilated to machine learning use iterative methods to solve the firstorder condition. When the parametric model to be calibrated is indeed convex and semidifferentiable, it is possible to use, for example, the stochastic gradient descent method as suggested by Bottou (2010). This last one allows to free oneself at each iteration from the calculation of the gradient on each observation of our learning base. Rather than making an average descent at each iteration, we start by drawing (without replacement) an observation \mathbf{x}_i among the n available. The model parameters are then corrected so that the prediction made from \mathbf{x}_i is as close as possible to the true value y_i. The method is then repeated until all the data have been reviewed. In this algorithm there is therefore as much iteration as there are observations. Unlike the gradient descent algorithm (or Newton’s method) at each iteration, only one gradient vector is calculated (and no longer n). However, it is sometimes necessary to run this algorithm several times to increase the convergence of the model parameters. If the objective is, for example, to minimize a loss function \ell between the estimator m_\beta (\mathbf{x}) and y (like the quadratic loss function, as in the Gaussian linear regression) the algorithm can be summarized as follows:
This algorithm can be repeated several times as a whole depending on the user’s needs. The advantage of this method is that at each iteration, it is not necessary to calculate the gradient on all observations (more sum). It is therefore suitable for large databases. This algorithm is based on a convergence in probability towards a neighborhood of the optimum (and not the optimum itself).
(references will be given in the very last post of that series) To be continued…
]]>Let’s define the scalar product in \mathbb{R}^n, ⟨\mathbf{a},\mathbf{b}⟩=\mathbf{a}^T\mathbf{b}, and let’s note \\cdot\ the associated Euclidean standard, \\mathbf{a}\=\sqrt{\mathbf{a}^T\mathbf{a}} (denoted \\cdot\_{\ell_2} in the next post). Note \mathcal{E}_X the space generated by all linear combinations of the \mathbf{X} components (adding the constant). If the explanatory variables are linearly independent, \mathbf{X} is a full (column) rank matrix and \mathcal{E}_X is a space of dimension p+1. Let’s assume from now on that the variables \mathbf{x} and y are centered here. Note that no law hypothesis is made in this section, the geometric properties are derived from the properties of expectation and variance in the set of finite variance variables.
With this notation, it should be noted that the linear model is written m(\mathbf{x})=⟨\mathbf{x},\beta⟩. The space H_z=\{\mathbf{x}\in\mathbb{R}^{p+1}:m(\mathbf{x})=z\} is a hyperplane (affine) that separates the space in two. Let’s define the orthogonal projection operator on \mathcal{E}_X, \Pi_X =\mathbf{X}(\mathbf{X}^T\mathbf{X})^{1} \mathbf{X}^T. Thus, the forecast that can be made for it is: \widehat{\mathbf{y}}=\mathbf{X}(\mathbf{X}^T\mathbf{X})^{1} \mathbf{X}^T\mathbf{y}=\Pi_X\mathbf{y}. As, \widehat{\varepsilon}=\mathbf{y}\widehat{\mathbf{y}}=(\mathbb{I}\Pi_X)\mathbf{y}=\Pi_{X^\perp}\mathbf{y}, we note that \widehat{\varepsilon}\perp\mathbf{x}, which will be interpreted as meaning that residuals are a term of innovation, unpredictable in the sense that \Pi_{X }\widehat{\varepsilon}=\mathbf{0}. The Pythagorean theorem is written here: \Vert \mathbf{y} \Vert^2=\Vert \Pi_{ {X}}\mathbf{y} \Vert^2+\Vert \Pi_{ {X}^\perp}\mathbf{y} \Vert^2=\Vert \Pi_{ {X}}\mathbf{y}\Vert^2+\Vert \mathbf{y}\Pi_{ {X}}\mathbf{y}\Vert^2=\Vert\widehat{\mathbf{y}}\Vert^2+\Vert\widehat{\mathbf{\varepsilon}}\Vert^2which is classically translated in terms of the sum of squares: \underbrace{\sum_{i=1}^n y_i^2}_{n\times\text{total variance}}=\underbrace{\sum_{i=1}^n \widehat{y}_i^2}_{n\times\text{explained variance}}+\underbrace{\sum_{i=1}^n (y_i\widehat{y}_i)^2}_{n\times\text{residual variance}} The coefficient of determination, R^2, is then interpreted as the square of the cosine of the angle \theta between \mathbf{y} and \Pi_X \mathbf{y} : R^2=\frac{\Vert \Pi_{{X}} \mathbf{y}\Vert^2}{\Vert \mathbf{y}\Vert^2}=1\frac{\Vert \Pi_{ {X}^\perp} \mathbf{y}\Vert^2}{\Vert \mathbf {y}\Vert^2}=\cos^2(\theta)An important application was obtained by Frish & Waugh (1933), when the explanatory variables are divided into two groups, \mathbf{X}=[\mathbf{X}_1 \mathbf{X}_2], so that the regression becomes y=\beta_0+\mathbf{X}_1 β_1+\mathbf{X}_2 β_2+\varepsilon. Frish & Waugh (1933) showed that two successive projections could be considered. Indeed, if \mathbf{y}_2^\star=\Pi_{X_1^\perp} \mathbf{y} and X_2^\star=\Pi_{X_1^\perp}\mathbf{X}_2, we can show that \widehat{\beta} _2=[{\mathbf{X}_2^\star}^T \mathbf{X}_2^\star]^{1}{\mathbf{X}_2^\star}^T \mathbf{y}_2^\star In other words, the overall estimate is equivalent to the combination of independent estimates of the two models if \mathbf{X}_2^\star=\mathbf{X}_2, i.e. \mathbf{X}_2\in \mathcal{E}_{X_1}^\perp, which can be noted \mathbf{x}_1\perp\mathbf{x}_2 We obtain here the FrischWaugh theorem which guarantees that if the explanatory variables between the two groups are orthogonal, then the overall estimate is equivalent to two independent regressions, on each of the sets of explanatory variables. This is a theorem of double projection, on orthogonal spaces. Many results and interpretations are obtained through geometric interpretations (fundamentally related to the links between conditional expectation and the orthogonal projection in space of variables of finite variance).
This geometric interpretation might help to get a better understanding of the problem of underidentification, i.e. the case where the real model would be y_i=\beta_0+ \mathbf{x}_1^T \beta_1+\mathbf{x}_2^T \beta_2+\varepsilon_i, but the estimated model is y_i=b_0+\mathbf{x}_1^T \mathbf{b}_1+\eta_i. The maximum likelihood estimator of \mathbf{b}_1 is \widehat{\mathbf{b}}_1=\mathbf {\beta}_1 + \underbrace{ (\mathbf {X}_1^T\mathbf {X}_1)^{1} \mathbf {X}_1^T \mathbf {X}_{2} \mathbf{\beta}_2}_{\mathbf{\beta}_{12}}+\underbrace{(\mathbf{X}_1^{T}\mathbf{X}_1)^{1} \mathbf{X}_1^T\varepsilon}_{\nu}so that \mathbb{E}[\widehat{\mathbf{b}}_1]=\beta_1+\beta_{12}, the bias ( \beta_{12}) being null only in the case where \mathbf{X}_1^T \mathbf{X}_2=\mathbf{0} (i. e. \mathbf{X}_1\perp \mathbf{X}_2 ): we find here a consequence of the FrischWaugh theorem.
On the other hand, overidentification corresponds to the case where the real model would be y_i=\beta_0+\mathbf{x}_1^T \beta_1+\varepsilon_i, but the estimated model is y_i=b_0+ \mathbf{x}_1^T \mathbf{b} _1+\mathbf{x}_2^T \mathbf{b}_2+\eta_i. In this case, the estimate is unbiased, in the sense that \mathbb{E}[\widehat{\mathbf{b}}_1]=\beta_1 but the estimator is not efficient. Later on, we will discuss an effective method for selecting variables (and avoid overidentification).
We can rewrite equation (4) in the form \widehat{\mathbf{y}}=\Pi_X\mathbf{y} which helps us to see the forecast directly as a linear transformation of the observations. More generally, a linear predictor can be obtained by considering m(\mathbf{x})=\mathbf{s}_{\mathbf{x}}^T \mathbf{y}, where \mathbf{s}_{\mathbf{x}} is a weight vector, which depends on \mathbf{x}, interpreted as a smoothing vector. Using the vectors \mathbf{s}_{\mathbf{x}_i}, calculated from the observations \mathbf{x}_i, we obtain a matrix \mathbf{S} of size n\times n, and \widehat{\mathbf{y}}=\mathbf{S}\mathbf{y}. In the case of the linear regression described above, \mathbf{s}_{\mathbf{x}}=\mathbf{X}[\mathbf{X}^T\mathbf{X}]^{1}\mathbf{x}, and in that case \text{trace}(\mathbf{S}) is the number of columns in the \mathbf{X} matrix (the number of explanatory variables). In this context of more general linear predictors, \text{trace}(\mathbf{S}) is often seen as equivalent to the number of parameters (or complexity, or dimension, of the model), and \nu=n\text{trace}(\mathbf{S}) is then the number of degrees of freedom (see Ruppert et al., 2003; Simonoff, 1996). The principle of parsimony says that we should minimize this dimension (the trace of the matrix \mathbf{S}) as much as possible. But in the general case, this dimension is more to obtain, explicitely.
The estimator introduced by Nadaraya (1964) and Watson (1964), in the case of a simple nonparametric regression, is also written in this form since\widehat{m}_h(x)=\mathbf{s}_{x}^T\mathbf{y}=\sum_{i=1}^n \mathbf{s}_{x,i}y_iwhere\mathbf{s}_{x,i}=\frac{K_h(xx_i)}{K_h(xx_1)+\cdots+K_h(xx_n)} where K(\cdot) is a kernel function, which assigns a value that is lower the closer x_i is to x, and h>0 is the bandwidth. The introduction of this metaparameter h is an important issue, as it should be chosen wisely. Using asymptotic developments, we can show that if X has density f, \text{biais}[\widehat{m}_h(x)]=\mathbb{E}[\widehat{m}_h(x)]m(x)\sim {h^2}\left(\frac{C_1 }{2}m''(x)+C_2 m'(x)\frac{f'(x)}{f(x)}\right)and \displaystyle{{\text{Var}[\widehat{m}_h(x)]\sim\frac{C_3}{{nh}}\frac{\sigma(x)}{f(x)}}}for some constants that can be estimated (see Simonoff (1996) for a discussion). These two functions evolve inversely with h, as shown in Figure 1 (where the metaparameter on the xaxis is here, actually, h^{1}). Keep in ming that we will see a similar graph in the context of machine learning models.
Figure 1. Choice of metaparameter and the Goldilocks problem: it must not be too large (otherwise there is too much variance), nor too small (otherwise there is too much bias).
The natural idea is then to try to minimize the mean square error, the MSE, defined as bias[\widehat{m}_h (x)]^2+Var[\widehat{m}_h (x)], and them integrate over x, which gives an optimal value for h of the form h^\star=O(n^{1/5}) , and reminds us of Silverman’s rule – see Silverman (1986). In larger dimensions, for continuous \mathbf{x} variables, a multivariate kernel with matrix bandwidth \mathbf{H} can be used, and \mathbb{E}[\widehat{m}_{\mathbf{H}}(\mathbf{x})]\sim m(\mathbf{x})+\frac{C_1}{2}\text{trace}\big(\mathbf{H}^Tm''(\mathbf{x})\mathbf{H}\big)+C_2\frac{m'(\boldsymbol{x})^T\mathbf{H}\mathbf{H}^T \nabla f(\mathbf{x})}{f(\mathbf{x})}while\text{Var}[\widehat{m}_{\mathbf{H}}(\mathbf{x})]\sim\frac{C_3}{n~\text{det}(\mathbf{H})}\frac{\sigma(\mathbf{x})}{f(\mathbf{x})}
If \mathbf{H} is a diagonal matrix, with the same term h on the diagonal, then h^\star=O(n^{1/(4+dim(\mathbf{x}))}. However, in practice, there will be more interest in the integrated version of the quadratic error, MISE(\widehat{m}_{h})=\mathbb{E}[MSE(\widehat{m}_{h}(X))]=\int MSE(\widehat{m}_{h}(x))dF(x)and we can prove that MISE[\widehat{m}_h]\sim \overbrace{\frac{h^4}{4}\left(\int x^2k(x)dx\right)^2\int\big[m''(x)+2m'(x)\frac{f'(x)}{f(x)}\big]^2dx}^{\text{bias}^2} +\overbrace{\frac{\sigma^2}{nh}\int k^2(x)dx \cdot\int\frac{dx}{f(x)}}^{\text{variance}}as n→∞ and nh→∞. Here we find an asymptotic relationship that again recalls Silverman’s (1986) order of magnitude, h^\star =n^{\frac{1}{5}}\left(\frac{C_1\int \frac{dx}{f(x)}}{C_2\int \big[m''(x)+2m'(x)\frac{f'(x)}{f(x)}\big]dx}\right)^{\frac{1}{5}}The main problem here, in practice, is that many of the terms in the expression above are unknown. Automatic learning offers computational techniques, when the econometrician used to searching for asymptotic (mathematical) properties.
To be continued (references mentioned above are online here)…
]]>The importance of probabilistic models in economics is rooted in Working’s (1927) questions and the attempts to answer them in Tinbergen’s two volumes (1939). The latter have subsequently generated a great deal of work, as recalled by Duo (1993) in his book on the foundations of econometrics, and more particularly in the first chapter “The Probability Foundations of Econometrics”. It should be recalled that Trygve Haavelmo was awarded the Nobel Prize in Economics in 1989 for his “clarification of the foundations of the probabilistic theory of econometrics”. Because as Haavelmo (1944) (initiating a profound change in econometric theory in the 1930s, as recalled in Morgan’s Chapter 8 (1990)) showed, econometrics is fundamentally based on a probabilistic model, for two main reasons. First, the use of statistical quantities (or “measures”) such as means, standard errors and correlation coefficients for inferential purposes can only be justified if the process generating the data can be expressed in terms of a probabilistic model. Second, the probability approach is relatively general, and is particularly well suited to the analysis of “dependent” and “nonhomogeneous” observations, as they are often found on economic data.We will then assume that there is a probabilistic space (\Omega,\mathcal{F},\mathbb{P}) such that observations (y_i,\mathbf{x}_i) are seen as realizations of random variables (Y_i, \mathbf{X}_i) . In practice, however, we are not very interested in the joint law of the couple (Y, \mathbf{X}) : the law of \mathbf{X} is unknown, and it is the law of Y conditional on \mathbf{X} that will be interested in. In the following, we will note x a single observation, \mathbf{x} a vector of observations, X a random variable, and \mathbf{X} a random vector. Abusively, \mathbf{X} may also designate the matrix of individual observations (denoted \mathbf{x}_i), depending on the context.
As recalled in Vapnik’s (1998) introduction, inference in parametric statistics is based on the following belief: the statistician knows the problem to be analyzed well, in particular, he knows the physical law that generates the stochastic properties of the data, and the function to be found is written via a finite number of parameters[1]. To find these parameters, the maximum likelihood method is used. The purpose of the theory is to justify this approach (by discovering and describing its favorable properties). We will see that in learning, philosophy is very different, since we do not have a priori reliable information on the statistical law underlying the problem, nor even on the function we would like to approach (we will then propose methods to construct an approximation from the data at our disposal, as in (1998)). A “golden age” of parametric inference, from 1930 to 1960, laid the foundations for mathematical statistics, which can be found in all statistical textbooks, including today. As Vapnik (1998) states, the classical parametric paradigm is based on the following three beliefs:
In this section we will come back to the construction of the econometric paradigm, directly inspired by that of classical inferential statistics.
Linear econometrics has been constructed under the assumption of individual data, which amounts to assuming independent variables (Y_i, \mathbf{X}_i) (if it is possible to imagine temporal observations – then we would have a process (Y_t, \mathbf{X}_t) – but we will not discuss time series here). More precisely, we will assume that, conditionally to the explanatory variables \mathbf{X}_i, the variables Y_i are independent. We will also assume that these conditional laws remain in the same parametric family, but that the parameter is a function of \mathbf{x}. In the Gaussian linear model it is assumed that: (Y\vert \mathbf{X}=\mathbf{x})\overset{\mathcal{L}}{\sim}\mathcal{N}(\mu(\mathbf{x}),\sigma^2)~~~~ (1)where \mu(\mathbf{x})=\beta_0+\mathbf{x}^T\mathbf{\beta} and \mathbf{\beta}\in\mathbb{R}^{p}.
It is usually called a ‘linear’ model since \mathbb{E}[Y\vert \mathbf{X}=\mathbf{x}]=\beta_0+\mathbf{x}^T\mathbf{\beta} is a linear combination of covariates[2]. It is said to be a homoscedastic model if Var[Y\mathbf{X}=\mathbf{x}]=\sigma^2, where \sigma^2 is a positive constant. To estimate the parameters, the traditional approach is to use the Maximum Likelihood estimator, as initially suggested by Ronald Fisher. In the case of the Gaussian linear model, loglikelihood is written: \log\mathcal{L}(\beta_0, \mathbf{\beta},\sigma^2\vert \mathbf{y},\mathbf{x}) = \frac{n}{2}\log[2\pi\sigma^2]  \frac{1}{2\sigma^2}\sum_{i=1}^n (y_i\beta_0\mathbf{x}_i^T\mathbf{\beta})^2Note that the term on the right, measuring a distance between the data and the model, will be interpreted as deviance in generalized linear models. Then we will set: (\widehat{\beta}_0,\widehat{\mathbf{\beta}},\widehat{\sigma}^2)=\text{argmax}\left\lbrace\log\mathcal{L}(\beta_0, \mathbf{\beta},\sigma^2\vert \mathbf{y},\mathbf{x})\right\rbraceThe maximum likelihood estimator is obtained by minimizing the sum of the error squares (the socalled “least squares” estimator) that we will find in the “machine learning” approach.
The first order conditions allow to find the normal equations, whose matrix writing is \mathbf{X}^T[\mathbf{y}\mathbf{X}\mathbf{\beta}]=\mathbf{0}, which can also be written (\mathbf{X}^T \mathbf{X})\mathbf{\beta}=\mathbf{X}^T \mathbf{y}. If \mathbf{X} is a full (column) rank matrix, then we find the classical estimator:\widehat{\mathbf{\beta}}=(\mathbf{X}^T\mathbf{X})^{1}\mathbf{X}^T\mathbf{y}=\mathbf{\beta}+(\mathbf{X}^T\mathbf{X})^{1}\mathbf{X}^{1}\mathbf{\varepsilon}~~~(2)using residualbased writing (as often in econometrics), y=\mathbf{x}^T\mathbf{\beta}+\varepsilon. Gauss Markov’s theorem ensures that this estimator is the unbiased linear estimator with minimum variance. It can then be shown that \widehat{\mathbf{\beta}}\sim\mathcal{N}(\mathbf{\beta},\sigma^2(\mathbf{X}^T\mathbf{X})^{1}), and in particular, if we simply need the first two moments : \mathbb{E}[\widehat{\mathbf{\beta}}]=\mathbf{\beta}~~~Var[\widehat{\mathbf{\beta}}]=\sigma^2 [\mathbf{X}^T\mathbf{X}]^{1}In fact, the normality hypothesis makes it possible to make a link with mathematical statistics, but it is possible to construct this estimator given by equation (2) without that Gaussian assumption. Hence, if we assume that Y\mathbf{X} has the same distribution as \mathbf{x}^T\mathbf{\beta}+\varepsilon, where \mathbb{E}[\varepsilon]=0, Var[\varepsilon]=\sigma^2 and Cov[X_j,\varepsilon]=0 for all j, then \widehat{\mathbf{\beta}} is an unbiased estimator of \mathbf{\beta} with smallest variance[3] among unbiased linear estimators. Furthermore, if we cannot get normality at finite distance, asymptotically this estimator is Gaussian, with \sqrt{n}(\widehat{\mathbf{\beta}}\mathbf{\beta})\overset{\mathcal{L}}{\rightarrow}\mathcal{N}(\mathbf{0},\mathbf{\Sigma})as n\rightarrow\infty, for some matrix \mathbf{\Sigma}.
The condition of having a full rank \mathbf{X} matrix can be (numerically) strong in large dimensions. If it is not satisfied, (\mathbf{X}^T \mathbf{X})^{1}\mathbf{X}^T does not exist. If \mathbb{I} denotes the identity matrix, however, it should be noted that (\mathbf{X}^T \mathbf{X}+\lambda\mathbb{I})^{1}\mathbf{X}^T still exists, whatever \lambda>0. This estimator is called the ridge estimator of level \lambda (introduced in the 1960s by Hoerl (1962), and associated with a regularization studied by Tikhonov (1963)). This estimator naturally appears in a Bayesian econometric context.
It is not uncommon to introduce the linear model from the distribution of the residuals, as we mentioned earlier. Also, equation (1) is written as often: y_i=\beta_0+\mathbf{x}_i^T\mathbf{\beta}+\varepsilon_i~~~~(3)where \varepsilon_i’s are realizations of independent and identically distributed random variables (i.i.d.) from some \mathcal{N}(0,\sigma^2) distribution. With a vector notation, we will write \mathbf{\varepsilon}\overset{\mathcal{L}}{\sim}\mathcal{N}(\mathbf{0},\sigma^2\mathbb{I}) . The estimated residuals are defined as: \widehat{\varepsilon}_i =y_i[\widehat{\beta}_0+\mathbf{x}_i^T\widehat{\mathbf{\beta}}] Those (estimated) residuals are basic tools for diagnosing the relevance of the model.
An extension of the model described by equation (1) has been proposed to take into account a possible heteroscedastic character: (Y\vert \mathbf{X}=\mathbf{x})\overset{\mathcal{L}}{\sim}\mathcal{N}(\mu(\mathbf{x}),\sigma^2(\mathbf{x}))where \sigma^2(\mathbf{x}) is a positive function of the explanatory variables. This model can be rewritten as: y_i=\beta_0+\mathbf{x}_i^T\mathbf{\beta}+\sigma^2(\mathbf{x}_i)\cdot\varepsilon_iwhere residuals are always i.i.d., with unit variance, \varepsilon_i=\frac{y_i[\beta_0+\mathbf{x}_i^T\mathbf{\beta}]}{\sigma(\mathbf{x}_i)} While residuals based equations are popular in linear econometrics (when the dependent variable is continuous), it is no longer popular in counting models, or logistic regression.
However, writing using an error term (as in equation (3)) raises many questions about the representation of an economic relationship between two quantities. For example, it can be assumed that there is a relationship (linear to begin with) between the quantities of a traded good, q and its price p. This allows us to imagine a supply equationq_i=\beta_0+\beta_1 p_i+u_i(u_i being an error term) where the quantity sold depends on the price, but in an equally legitimate way, one can imagine that the price depends on the quantity produced (what one could call a demand equation), p_i=\alpha_0+\alpha_1 q_i+v_i(v_i denoting another error term). Historically, the error term in equation (3) could be interpreted as an idiosyncratic error on the variable y, the socalled explanatory variables being assumed to be fixed, but this interpretation often makes the link between an economic relationship and a complicated economic model difficult, the economic theory speaking abstractly about a relationship between a magnitude, the econometric model imposing a specific shape (what magnitude is y and what magnitude is x) as shown in more detail in Morgan (1990) Chapter 7.
(references mentioned above are online here). To be continued…
[1] This approach can be compared to structural econometrics, as presented for example in Kean (2010).
[2] Here, we will try to distinguish \beta_0, the intercept, and the other parameters \mathbf{\beta}, since they are considered differently in many extensions (e.g. regularization). Nevertheless, in many expressions \mathbf{\beta} will denote the joint vector (\beta_0, \mathbf{\beta}), for general formulas, to avoid too heavy notations.
[3] In the sense that the difference between variance matrices is a positive matrix.
]]>