Somewhere else, part 179

Some posts and articles worth reading, here and there

Continue reading Somewhere else, part 179

Extracting datasets from excel files in a zipped folder

The title of the post is a bit long, but that’s the problem I was facing this morning: importing dataset from files, online. I mean, it was not a “problem” (since I can always download, and extract manually the files), more a challenge (I should be able to do it in R, directly). The files are located on ressources-actuarielles.net, in a zip file. Those are mortality tables used in French speaking African countries, and I guess that one problem came from special characters, such as “é” or “è”… When you open the zip file, you see a folder

and in that folder, several files that I would like to import

Continue reading Extracting datasets from excel files in a zipped folder

Somewhere else, part 178

Some posts and articles worth reading,

Though other services, like Twitter and Google News, can also exert a large influence, Facebook is at the forefront of a fundamental change in how people consume journalism. Most readers now come to it not through the print editions of newspapers and magazines or their home pages online, but through social media and search engines driven by an algorithm, a mathematical formula that predicts what users might want to read. It is a world of fragments, filtered by code and delivered on demand. For news organizations, said Cory Haik, senior editor for digital news at The Washington Post, the shift represents “the great unbundling” of journalism. Just as the music industry has moved largely from selling albums to songs bought instantly online, publishers are increasingly reaching readers through individual pieces rather than complete editions of newspapers or magazines. A publication’s home page, said Edward Kim, a co-founder of SimpleReach, will soon be important more as an advertisement of its brand than as a destination for readers. “People won’t type in WashingtonPost.com anymore,” Ms. Haik said. “It’s search and social.” The shift raises questions about the ability of computers to curate news, a role traditionally played by editors. It also has broader implications for the way people consume information, and thus how they see the world.

Continue reading Somewhere else, part 178

Log-transform kernel density estimation of income distribution

Our paper Log-transform kernel density estimationof income distribution, written with Emmanuel Flachaire is now available on http://papers.ssrn.com/id=2514882,

Standard kernel density estimation methods are very often used in practice to estimate density function. It works well in numerous cases. However, it is known not to work so well with skewed, multimodal and heavy-tailed distributions. Such features are usual with income distributions, defined over the positive support. We first show that a preliminary logarithmic transformation of the data, combined with standard kernel density estimation methods, can provide a much better fit of the overall density estimation. Then, we show that the fit of the bottom of the distribution may not be satisfactory, even if a better fit of the upper tail can be obtained in general.

Somewhere else, part 177

Some posts and articles worth reading, here and there

Unknown

Continue reading Somewhere else, part 177

Tests basés sur la vraisemblance – score

Une autre grandeur intéressant est le score, qui est la dérivée de la vraisemblance. Intuitivement (c’est l’idée de la condition du premier ordre),   et  seront proches si les dérivées en ces points sont proches. En  la dérivée est nulle, donc on va se demander ici, tout simplement, si la dérivée en  est proche de 0. Ou pas.

Continue reading Tests basés sur la vraisemblance – score

Tests basés sur la vraisemblance – Rapport de Vraisemblance

Chose promise, chose due. J’avais dit qu’on parlerait du test de rapport de vraisemblance. L’idée – visuelle – est d’avoir une lecture dans l’autre sens : au lieu de se demander si  et  sont proches, on va se demander si   et  sont proches.

Si la fonction de vraisemblance est suffisamment régulière, on se pose la même question.

Continue reading Tests basés sur la vraisemblance – Rapport de Vraisemblance

Somewhere else, part 176

yes, extreme winds  cause a waterfall (in England) to blow upward

and some more posts and articles worth reading

Continue reading Somewhere else, part 176

Tests basés sur la vraisemblance – Wald

Petit rappel préliminaire. Si on dispose d’un échantillon , i.id de loi , où le paramètre  est inconnu, alors on peut calculer le maximum de la vraisemblance, ce qui nous donnera un estimateur intéressant (cf. cours de stat). Illustrons avec un jeu de pile ou face. Reprenons l’échantillon de mon précédent billet,

> X=c(0, 0, 1, 1, 0, 1, 1, 1, 1, 0,
      0, 0, 1, 0, 1, 0, 1, 1, 0, 1)

On peut ici tracer la fonction de vraisemblance

ou (un peu plus malin) la fonction de log-vraisemblance

> p=seq(0,1,by=.01)
> logL=function(p)
+ {sum(log(dbinom(X,size=1,prob=p)))}
> plot(p,Vectorize(logL)(p),
+ type="l",col="red",lwd=2)
> p0=.5
> abline(v=p0,col="blue")

La valeur correspondant au trait bleu correspondant au cas que l’on va chercher à tester, de pièce non pipée, soit  (de manière très générale)

Continue reading Tests basés sur la vraisemblance – Wald

Les tests et la logique, modus tollens

Quand on apprend la logique, on apprend la notion de modus tollens, correspondant à une notion de contraposition. Si on a une proposition du genre , alors la proposition contraposée est .  Et on apprend que les deux propositions sont équivalentes (je fais de la logique classique). Par exemple si  correspond à “feu” et  à “fumée“,  signifie que tout feu fait de la fumée. Si cette affirmation est vrai, alors il n’y a pas de fumée sans feu, et donc , autrement dit, s’il n’y a pas de fumée, c’est qu’il n’y a pas de feu.

Continue reading Les tests et la logique, modus tollens

Test du chi-deux et indépendance

Considérons le tableau de contingence suivant

> N=margin.table(HairEyeColor, c(1,2))
> N
       Eye
Hair    Brown Blue Hazel Green
  Black    68   20    15     5
  Brown   119   84    54    29
  Red      26   17    14    14
  Blond     7   94    10    16

avec ici les comptages, que l’on peut aussi traduire sous la forme de probabilités jointes (empiriques)

> N/n
       Eye
Hair    Brown  Blue Hazel Green
  Black 0.115 0.034 0.025 0.008
  Brown 0.201 0.142 0.091 0.049
  Red   0.044 0.029 0.024 0.024
  Blond 0.012 0.159 0.017 0.027

Continue reading Test du chi-deux et indépendance

Kernel Density Estimation with Ripley’s Circumferential Correction

The revised version of the paper Kernel Density Estimation with Ripley’s Circumferential Correction is now online, on hal.archives-ouvertes.fr/.

In this paper, we investigate (and extend) Ripley’s circumference method to correct bias of density estimation of edges (or frontiers) of regions. The idea of the method was theoretical and difficult to implement. We provide a simple technique — based of properties of Gaussian kernels — to efficiently compute weights to correct border bias on frontiers of the region of interest, with an automatic selection of an optimal radius for the method. We illustrate the use of that technique to visualize hot spots of car accidents and campsite locations, as well as location of bike thefts.

There are new applications, and new graphs, too

Most of the codes can be found on https://github.com/ripleyCorr/Kernel_density_ripley (as well as datasets).

Somewhere else, part 175

(Halloween Tree, by Glen Brogan)

Some posts and articles worth reading, here and there

A bank’s earnings are a quantum event; they are entirely probabilistic, and the answer you get depends on who’s doing the observing. You make some guesses with some degree of statistical likelihood, and then you apply one of a half-dozen accounting regimes to the guesses, and you get a number, and then you’re like, ooh, look at this number, it’s so numeric.

Continue reading Somewhere else, part 175

Removing Uncited References in a Tex File (with R)

Last week, with @3wen, we were working a the revised version of our work on smoothing densities of spatial processes (with edge correction). Usually, once you have revised the paper, some references were added, others were droped. But you need to spend some time, to check that all references are actually mentioned in the paper. For instance, consider the following compiled tex file :

Continue reading Removing Uncited References in a Tex File (with R)

Somewhere else, part 174

Some posts and articles worth reading

Continue reading Somewhere else, part 174