Benford law and lognormal distributions

March 28, 2013
By

Benford’s law is nowadays extremely popular (see e.g. http://en.wikipedia.org/…). It is usually claimed that, for a given set data set, changing units does not affect the distribution of the first digit. Thus, it should be related to scale invariant distributions. Heuristically, scale (or unit) invariance means that the density of the measure http://latex.codecogs.com/gif.latex?%20X (or probability function) http://latex.codecogs.com/gif.latex?f(x) should be proportional to http://latex.codecogs.com/gif.latex?f(kx). Thus, because densities integrate to 1, the proportionality coefficient has to be http://latex.codecogs.com/gif.latex?k^{-1}, and therefore, http://latex.codecogs.com/gif.latex?f should satisfy the following functional equation, http://latex.codecogs.com/gif.latex?%20kf(kx)=f(x), for all http://latex.codecogs.com/gif.latex?%20x in http://latex.codecogs.com/gif.latex?%20(1,\infty) and http://latex.codecogs.com/gif.latex?%20k in http://latex.codecogs.com/gif.latex?%20(0,\infty). The solution of this functional equation is http://latex.codecogs.com/gif.latex?%20f(x)=x^{-1}, I guess this can be proved easily solving ordinary differential equation

http://latex.codecogs.com/gif.latex?%20\frac{d}{dk}%20(kf(kx))=0

Now if http://latex.codecogs.com/gif.latex?%20D denotes the first digit of http://latex.codecogs.com/gif.latex?%20X, in base 10, then

http://latex.codecogs.com/gif.latex?%20\mathbb{P}(D=d)=\frac{\displaystyle{\int_d^{d+1}%20f(x)dx}}{{\displaystyle{\int_1^{10}%20f(x)dx}}}=\cdots=\frac{\displaystyle{\log\left(1+\frac{1}{d}\right)}}{\log(10)}Which is the so-called Benford’s law. So, this distribution looks like that

> (benford=log(1+1/(1:9))/log(10))
[1] 0.30103000 0.17609126 0.12493874 0.09691001 0.07918125 
[6] 0.06694679 0.05799195 0.05115252 0.04575749
> names(benford)=1:9
> sum(benford)
[1] 1
> barplot(benford,col="white",ylim=c(-.045,.3))
> abline(h=0)

To compute the empirical distribution from a sample, use the following function

> firstdigit=function(x){
+ if(x>=1){x=as.numeric(substr(as.character(x),1,1)); zero=FALSE}
+ if(x<1){zero=TRUE}
+ while(zero==TRUE){
+ x=x*10; zero=FALSE
+ if(trunc(x)==0){zero=TRUE}
+ }
+ return(trunc(x))
+ }

and then

> Xd=sapply(X,firstdigit)
> table(Xd)/1000

In Benford’s Law: An Empirical Investigation and a Novel Explanation, we can read

It is not a mathematical article, so do not expect any formal proof in this paper. At least, we can run monte carlo simulation, and see what’s going on if we generate samples from a lognormal distribution with variance http://latex.codecogs.com/gif.latex?%20\sigma^2. For instance, with a unit variance,

> set.seed(1)
> s=1
> X=rlnorm(n=1000,0,s)
> Xd=sapply(X,firstdigit)
> table(Xd)/1000
Xd
    1     2     3     4     5     6     7     8     9 
0.288 0.172 0.121 0.086 0.075 0.072 0.073 0.053 0.060 
> T=rbind(benford,-table(Xd)/1000)
> barplot(T,col=c("red","white"),ylim=c(-.045,.3))
> abline(h=0)

Clearly, it not far away from Benford’s law. Perhaps a more formal test can be considered, for instance Pearson’s http://latex.codecogs.com/gif.latex?%20\chi^2 (goodness of fit) test.

> chisq.test(T,p=benford)

	Chi-squared test for given probabilities

data:  T 
X-squared = 10.9976, df = 8, p-value = 0.2018

So yes, Benford’s law is admissible ! Now, if we consider the case where http://latex.codecogs.com/gif.latex?%20\sigma is smaller (say 0.9), it is a rather different story,

compared with the case where http://latex.codecogs.com/gif.latex?%20\sigma is larger (say 1.1)

It is possible to generate several samples (always the same size, here 1,000 observations), just change the variance parameter http://latex.codecogs.com/gif.latex?%20\sigma and compute the http://latex.codecogs.com/gif.latex?%20p-value of the test. There might be one tricky part: when generating samples from lognormal distributions with small variance, it might be possible that some digits do not appear at all. On that case, there is a problem with the test. So we just use here

> T=table(Xd)
> T=T[as.character(1:9)]
> T[is.na(T)]=0
> PVAL[i]=chisq.test(T,p=benford)$p.value

Boxplots of the http://latex.codecogs.com/gif.latex?%20p-value of the test are the following,

When http://latex.codecogs.com/gif.latex?%20\sigma is too small, it is clearly not Benford’s distribution: for half (or more) of our samples, the http://latex.codecogs.com/gif.latex?%20p-value is lower than 5%. On the other hand, when http://latex.codecogs.com/gif.latex?%20\sigma is large (enough), Benford’s distribution is the distribution of the first digit of lognormal samples, since 95% of our samples have  http://latex.codecogs.com/gif.latex?%20p-values higher than 5% (and the distribution of the http://latex.codecogs.com/gif.latex?%20p-value is almost uniform on the unit interval). Here is the proportion of samples where the http://latex.codecogs.com/gif.latex?%20p-value was lower than 5% (on 5,000 generations each time)

Note that it is also possible to compute the http://latex.codecogs.com/gif.latex?%20p-value of Komogorov-Smirnov test, testing if the http://latex.codecogs.com/gif.latex?%20p-value has a uniform distribution,

> ks.test(PVAL[,s], "punif")$p.value

Indeed, if http://latex.codecogs.com/gif.latex?%20\sigma is larger than 1.15 (around that value), it looks like Benford’s law is a suitable distribution for the first digit.


Print This Post Print This Post

Tags: , , ,

Posted in: Computer, Statistics

10 Responses to “ Benford law and lognormal distributions ”

  1. Govind S. Mudholkar on 19/09/2013 at 8:19 PM

    BENFORD’S LAW FOR COEFFICIENTS OF MODULAR FORMS …
    http://www.ams.org/proc/2011-139-05/…/S0002-9939-2010-10577-4.pdf‎
    by T Anderson – ‎2011 – ‎Cited by 1 – ‎Related articles
    Oct 5, 2010 – PROCEEDINGS OF THE. AMERICAN MATHEMATICAL SOCIETY. Volume … Here we prove that Benford’s law holds for coefficients of an in-.
    Previous article – Proceedings of the American Mathematical Society
    http://www.ams.org/proc/2011-139-05/S0002-9939-2010-10577-4/‎
    by T Anderson – ‎2011 – ‎Cited by 1 – ‎Related articles
    Benford’s law for coefficients of modular forms and partition functions … Abstract: Here we prove that Benford’s law holds for coefficients of an infinite class of …
    Simon Newcomb and “Natural Numbers” (Benford’s Law) – American …
    http://www.ams.org/samplings/feature-column/fcarc-newcomb‎
    by S Newcomb – ‎Related articles
    tony at math.sunysb.edu … This law was rediscovered by Frank Benford (“The law of anomalous numbers,” …. Frank Benford, The law of anomalous numbers, Proceedings of the American Philosophical Society 78 (1938) 551-572. Theodore P. Hill, Base-invariance implies Benford’s law, Proceedings of the A. M. S. 123 …
    Base-invariance implies Benford’s law – American Mathematical …
    http://www.ams.org/proc/1995-123-03/S0002-9939-1995-1233974-8/‎
    by TP Hill – ‎1995 – ‎Cited by 150 – ‎Related articles
    Proceedings of the American Mathematical Society. Journals … Abstract: A derivation of Benford’s Law or the First-Digit Phenomenon is given assuming only …
    Works that this work references – Benford Online Bibliog

  2. NicolasGauvrit on 16/04/2013 at 2:09 AM

    Thanks Peter, this is exactly the book I should have read before !

    • Peter Vijn on 16/04/2013 at 7:02 AM

      Nicholas,
      The order in which we digest doesn’t matter, as long as the full picture emerges. For Benford the full picture is that it naturally arises from any wide distribution, defined as sigma > 0.45 on the (10)log scale, which reduces it to merely an epiphenomenon.
      With this knowledge, the popular game of computing and publishing isolated Benford distributions is silly and should best be avoided. If Benford is of interest for a particular dataset, a band of distributions (consisting as the overlay of many Benford distributions) should be plotted instead, obtained by multiplying the original dataset with numbers between 1 and 10 in small increments (e.g. by iterative multiplication with say 1.05) and computing Benford with every step. Essentially, the ‘ones scaling test’ is applied to all nine first digits simultaneously which is a much better basis for further analysis and discussion. But even when using this ‘advanced’ method, all that is accomplished is a inefficient test for the width of the distribution of the original data.
      Cheers, Peter

  3. Peter Vijn on 10/04/2013 at 11:21 AM

    Thanks for this! Very nice. I strongly recommend Steven Smith’s DSP chapter on Benford’s law (http://www.dspguide.com/ch34.htm 1997). After reading this the mystery is fully solved and explained. Your observations in these simulations are exactly in line with this, and there are also no surprises in Nicolas’ paper mentioned above.

  4. NicolasGauvrit on 02/04/2013 at 8:07 AM

    You should be interested in two papers we wrote (in french) some time ago. One proves a theorem from which Benford may be derived, and the second shows that the “generalized” Benford’s law stating that Frac(f(X))~Unif mod 1 is actually more often true than the classical Benford’s law — for some f.
    http://www.ehess.fr/revue-msh/pdf/N186R1370.pdf
    http://msh.revues.org/10363?file=1

    We also wrote a chapter (in English) about the same subject: http://arxiv.org/abs/0910.1359

    Cheers,

  5. datanalytics » Las leyes de Benford on 02/04/2013 at 2:53 AM

    [...] bien, Arthur Charpentier ha escrito sobre la Ley de Benford en su bitácora Freakonometrics recientemente explicando bajo qué condiciones son aplicables las probabilidades de los dígitos [...]

  6. Carlos J. Gil Bellosta on 31/03/2013 at 3:44 PM

    I wrote a critical review of Benford’s Law some time ago in my blog pointing precisely at the magical way in which the first digit probabilities are presented. You can see it at

    http://www.datanalytics.com/blog/2011/09/15/la-ley-de-benford/

    In fact, there does not seem to be a “single” Benford Law but several depending on the distribution of your data. As you point out, it may be the case that for some of those other distributions Benford’s probabilities are a good proxy, though.

    • Arthur Charpentier on 31/03/2013 at 7:08 PM

      Exactly ! I’d be glad to see such a “generalized Benford law” which might work when data have regularly varying (Pareto type) tails, and a more uniform distribution. Benford is one possible distribution, which works well with the power decay. But alternative distribution should be possible for other underlying distributions, as you mention.

  7. DM on 29/03/2013 at 5:01 AM

    I can’t really understand what the fuss is over Benford’s law personally. At least things like the Pareto law let you focus on smaller subsets and concentrate your efforts.

    This aside, is there more power in reviewing the link between the coefficient of variation of the lognormal vs Benford’s law – ie the CoV encompasses all of the lognormal variability in a scale invariant metric?

  8. Christopher D. Long on 29/03/2013 at 4:48 AM

    Fix f(1) in your functional equation, then setting x=1 we get k*f(k) = f(1) for all k, thus f(k) = f(1)/k for all k.

Leave a Reply

Categories

Students

Keep calm…

Keep calm…

Research blogs