# Somewhere else, part 179

Some posts and articles worth reading, here and there

# Extracting datasets from excel files in a zipped folder

The title of the post is a bit long, but that’s the problem I was facing this morning: importing dataset from files, online. I mean, it was not a “problem” (since I can always download, and extract manually the files), more a challenge (I should be able to do it in R, directly). The files are located on ressources-actuarielles.net, in a zip file. Those are mortality tables used in French speaking African countries, and I guess that one problem came from special characters, such as “é” or “è”… When you open the zip file, you see a folder

and in that folder, several files that I would like to import

# Somewhere else, part 178

Some posts and articles worth reading,

# Log-transform kernel density estimation of income distribution

Our paper Log-transform kernel density estimationof income distribution, written with Emmanuel Flachaire is now available on http://papers.ssrn.com/id=2514882,

Standard kernel density estimation methods are very often used in practice to estimate density function. It works well in numerous cases. However, it is known not to work so well with skewed, multimodal and heavy-tailed distributions. Such features are usual with income distributions, defined over the positive support. We first show that a preliminary logarithmic transformation of the data, combined with standard kernel density estimation methods, can provide a much better fit of the overall density estimation. Then, we show that the fit of the bottom of the distribution may not be satisfactory, even if a better fit of the upper tail can be obtained in general.

# Somewhere else, part 177

Some posts and articles worth reading, here and there

# Tests basés sur la vraisemblance – score

Une autre grandeur intéressant est le score, qui est la dérivée de la vraisemblance. Intuitivement (c’est l’idée de la condition du premier ordre),  $\widehat\theta_n$ et $\theta_0$ seront proches si les dérivées en ces points sont proches. En $\widehat\theta_n$ la dérivée est nulle, donc on va se demander ici, tout simplement, si la dérivée en $\theta_0$ est proche de 0. Ou pas.

# Tests basés sur la vraisemblance – Rapport de Vraisemblance

Chose promise, chose due. J’avais dit qu’on parlerait du test de rapport de vraisemblance. L’idée – visuelle – est d’avoir une lecture dans l’autre sens : au lieu de se demander si $\widehat\theta_n$ et $\theta_0$ sont proches, on va se demander si $\log\mathcal{L}(\widehat\theta_n)$  et $\log\mathcal{L}(\theta_0)$ sont proches.

Si la fonction de vraisemblance est suffisamment régulière, on se pose la même question.

# Somewhere else, part 176

yes, extreme winds  cause a waterfall (in England) to blow upward

# Tests basés sur la vraisemblance – Wald

Petit rappel préliminaire. Si on dispose d’un échantillon $X_i$, i.id de loi $F_{\boldsymbol{\theta}_\star}$, où le paramètre $\boldsymbol{\theta}_\star$ est inconnu, alors on peut calculer le maximum de la vraisemblance, ce qui nous donnera un estimateur intéressant (cf. cours de stat). Illustrons avec un jeu de pile ou face. Reprenons l’échantillon de mon précédent billet,

> X=c(0, 0, 1, 1, 0, 1, 1, 1, 1, 0,
0, 0, 1, 0, 1, 0, 1, 1, 0, 1)

On peut ici tracer la fonction de vraisemblance

$\theta\mapsto \mathcal{L}(\theta,\mathcal{X})=\theta^{\sum X_i}[1-\theta]^{n-\sum X_i}$

ou (un peu plus malin) la fonction de log-vraisemblance

$\theta\mapsto \log\mathcal{L}(\theta,\mathcal{X})$

> p=seq(0,1,by=.01)
> logL=function(p)
+ {sum(log(dbinom(X,size=1,prob=p)))}
> plot(p,Vectorize(logL)(p),
+ type="l",col="red",lwd=2)
> p0=.5
> abline(v=p0,col="blue")

La valeur correspondant au trait bleu correspondant au cas que l’on va chercher à tester, de pièce non pipée, soit $\theta_0$ (de manière très générale)

# Les tests et la logique, modus tollens

Quand on apprend la logique, on apprend la notion de modus tollens, correspondant à une notion de contraposition. Si on a une proposition du genre $A \Rightarrow B$, alors la proposition contraposée est $\text{non }B \Rightarrow \text{non }A$.  Et on apprend que les deux propositions sont équivalentes (je fais de la logique classique). Par exemple si $A$ correspond à “feu” et $B$ à “fumée“, $A \Rightarrow B$ signifie que tout feu fait de la fumée. Si cette affirmation est vrai, alors il n’y a pas de fumée sans feu, et donc $\text{non }B \Rightarrow \text{non }A$, autrement dit, s’il n’y a pas de fumée, c’est qu’il n’y a pas de feu.

# Test du chi-deux et indépendance

Considérons le tableau de contingence suivant

> N=margin.table(HairEyeColor, c(1,2))
> N
Eye
Hair    Brown Blue Hazel Green
Black    68   20    15     5
Brown   119   84    54    29
Red      26   17    14    14
Blond     7   94    10    16

avec ici les comptages, que l’on peut aussi traduire sous la forme de probabilités jointes (empiriques)

> N/n
Eye
Hair    Brown  Blue Hazel Green
Black 0.115 0.034 0.025 0.008
Brown 0.201 0.142 0.091 0.049
Red   0.044 0.029 0.024 0.024
Blond 0.012 0.159 0.017 0.027

# Kernel Density Estimation with Ripley’s Circumferential Correction

The revised version of the paper Kernel Density Estimation with Ripley’s Circumferential Correction is now online, on hal.archives-ouvertes.fr/.

In this paper, we investigate (and extend) Ripley’s circumference method to correct bias of density estimation of edges (or frontiers) of regions. The idea of the method was theoretical and difficult to implement. We provide a simple technique — based of properties of Gaussian kernels — to efficiently compute weights to correct border bias on frontiers of the region of interest, with an automatic selection of an optimal radius for the method. We illustrate the use of that technique to visualize hot spots of car accidents and campsite locations, as well as location of bike thefts.

There are new applications, and new graphs, too

Most of the codes can be found on https://github.com/ripleyCorr/Kernel_density_ripley (as well as datasets).

# Somewhere else, part 175

(Halloween Tree, by Glen Brogan)

Some posts and articles worth reading, here and there

A bank’s earnings are a quantum event; they are entirely probabilistic, and the answer you get depends on who’s doing the observing. You make some guesses with some degree of statistical likelihood, and then you apply one of a half-dozen accounting regimes to the guesses, and you get a number, and then you’re like, ooh, look at this number, it’s so numeric.

# Removing Uncited References in a Tex File (with R)

Last week, with @3wen, we were working a the revised version of our work on smoothing densities of spatial processes (with edge correction). Usually, once you have revised the paper, some references were added, others were droped. But you need to spend some time, to check that all references are actually mentioned in the paper. For instance, consider the following compiled tex file :