Benford’s law is nowadays extremely popular (see e.g. http://en.wikipedia.org/…). It is usually claimed that, for a given set data set, changing units *does not *affect the distribution of the first digit. Thus, it should be related to scale invariant distributions. Heuristically, scale (or unit) invariance means that the density of the measure (or probability function) should be proportional to . Thus, because densities integrate to 1, the proportionality coefficient has to be , and therefore, should satisfy the following functional equation, , for all in and in . The solution of this functional equation is , I guess this can be proved easily solving ordinary differential equation

Now if denotes the first digit of , in base 10, then

Which is the so-called *Benford’s law*. So, this distribution looks like that

> (benford=log(1+1/(1:9))/log(10)) [1] 0.30103000 0.17609126 0.12493874 0.09691001 0.07918125 [6] 0.06694679 0.05799195 0.05115252 0.04575749 > names(benford)=1:9 > sum(benford) [1] 1 > barplot(benford,col="white",ylim=c(-.045,.3)) > abline(h=0)

To compute the empirical distribution from a sample, use the following function

> firstdigit=function(x){ + if(x>=1){x=as.numeric(substr(as.character(x),1,1)); zero=FALSE} + if(x<1){zero=TRUE} + while(zero==TRUE){ + x=x*10; zero=FALSE + if(trunc(x)==0){zero=TRUE} + } + return(trunc(x)) + }

and then

> Xd=sapply(X,firstdigit) > table(Xd)/1000

In Benford’s Law: An Empirical Investigation and a Novel Explanation, we can read

It is not a mathematical article, so do not expect any formal *proof* in this paper. At least, we can run monte carlo simulation, and see what’s going on if we generate samples from a lognormal distribution with variance . For instance, with a unit variance,

> set.seed(1) > s=1 > X=rlnorm(n=1000,0,s) > Xd=sapply(X,firstdigit) > table(Xd)/1000 Xd 1 2 3 4 5 6 7 8 9 0.288 0.172 0.121 0.086 0.075 0.072 0.073 0.053 0.060 > T=rbind(benford,-table(Xd)/1000) > barplot(T,col=c("red","white"),ylim=c(-.045,.3)) > abline(h=0)

Clearly, it not far away from Benford’s law. Perhaps a more formal test can be considered, for instance Pearson’s (goodness of fit) test.

> chisq.test(T,p=benford) Chi-squared test for given probabilities data: T X-squared = 10.9976, df = 8, p-value = 0.2018

So yes, Benford’s law is admissible ! Now, if we consider the case where is smaller (say 0.9), it is a rather different story,

compared with the case where is larger (say 1.1)

It is possible to generate several samples (always the same size, here 1,000 observations), just change the variance parameter and compute the -value of the test. There might be one tricky part: when generating samples from lognormal distributions with small variance, it might be possible that some digits do not appear at all. On that case, there is a problem with the test. So we just use here

> T=table(Xd) > T=T[as.character(1:9)] > T[is.na(T)]=0 > PVAL[i]=chisq.test(T,p=benford)$p.value

Boxplots of the -value of the test are the following,

When is too small, it is clearly not Benford’s distribution: for half (or more) of our samples, the -value is lower than 5%. On the other hand, when is large (enough), Benford’s distribution is *the* distribution of the first digit of lognormal samples, since 95% of our samples have -values higher than 5% (and the distribution of the -value is almost uniform on the unit interval). Here is the proportion of samples where the -value was lower than 5% (on 5,000 generations each time)

Note that it is also possible to compute the -value of Komogorov-Smirnov test, testing if the -value has a uniform distribution,

> ks.test(PVAL[,s], "punif")$p.value

Indeed, if is larger than 1.15 (around that value), it looks like Benford’s law is a suitable distribution for the first digit.

BENFORD’S LAW FOR COEFFICIENTS OF MODULAR FORMS …

http://www.ams.org/proc/2011-139-05/…/S0002-9939-2010-10577-4.pdf

by T Anderson – 2011 – Cited by 1 – Related articles

Oct 5, 2010 – PROCEEDINGS OF THE. AMERICAN MATHEMATICAL SOCIETY. Volume … Here we prove that Benford’s law holds for coefficients of an in-.

Previous article – Proceedings of the American Mathematical Society

http://www.ams.org/proc/2011-139-05/S0002-9939-2010-10577-4/

by T Anderson – 2011 – Cited by 1 – Related articles

Benford’s law for coefficients of modular forms and partition functions … Abstract: Here we prove that Benford’s law holds for coefficients of an infinite class of …

Simon Newcomb and “Natural Numbers” (Benford’s Law) – American …

http://www.ams.org/samplings/feature-column/fcarc-newcomb

by S Newcomb – Related articles

tony at math.sunysb.edu … This law was rediscovered by Frank Benford (“The law of anomalous numbers,” …. Frank Benford, The law of anomalous numbers, Proceedings of the American Philosophical Society 78 (1938) 551-572. Theodore P. Hill, Base-invariance implies Benford’s law, Proceedings of the A. M. S. 123 …

Base-invariance implies Benford’s law – American Mathematical …

http://www.ams.org/proc/1995-123-03/S0002-9939-1995-1233974-8/

by TP Hill – 1995 – Cited by 150 – Related articles

Proceedings of the American Mathematical Society. Journals … Abstract: A derivation of Benford’s Law or the First-Digit Phenomenon is given assuming only …

Works that this work references – Benford Online Bibliog

Please look for “Scale invariance implies Benford’s law” in the Proceedings of American Mathematical Society. I do not remember the author or year.

GM

Thanks Peter, this is exactly the book I should have read before !

Nicholas,

The order in which we digest doesn’t matter, as long as the full picture emerges. For Benford the full picture is that it naturally arises from any wide distribution, defined as sigma > 0.45 on the (10)log scale, which reduces it to merely an epiphenomenon.

With this knowledge, the popular game of computing and publishing isolated Benford distributions is silly and should best be avoided. If Benford is of interest for a particular dataset, a band of distributions (consisting as the overlay of many Benford distributions) should be plotted instead, obtained by multiplying the original dataset with numbers between 1 and 10 in small increments (e.g. by iterative multiplication with say 1.05) and computing Benford with every step. Essentially, the ‘ones scaling test’ is applied to all nine first digits simultaneously which is a much better basis for further analysis and discussion. But even when using this ‘advanced’ method, all that is accomplished is a inefficient test for the width of the distribution of the original data.

Cheers, Peter

Thanks for this! Very nice. I strongly recommend Steven Smith’s DSP chapter on Benford’s law (http://www.dspguide.com/ch34.htm 1997). After reading this the mystery is fully solved and explained. Your observations in these simulations are exactly in line with this, and there are also no surprises in Nicolas’ paper mentioned above.

You should be interested in two papers we wrote (in french) some time ago. One proves a theorem from which Benford may be derived, and the second shows that the “generalized” Benford’s law stating that Frac(f(X))~Unif mod 1 is actually more often true than the classical Benford’s law — for some f.

http://www.ehess.fr/revue-msh/pdf/N186R1370.pdf

http://msh.revues.org/10363?file=1

We also wrote a chapter (in English) about the same subject: http://arxiv.org/abs/0910.1359

Cheers,

I wrote a critical review of Benford’s Law some time ago in my blog pointing precisely at the magical way in which the first digit probabilities are presented. You can see it at

http://www.datanalytics.com/blog/2011/09/15/la-ley-de-benford/

In fact, there does not seem to be a “single” Benford Law but several depending on the distribution of your data. As you point out, it may be the case that for some of those other distributions Benford’s probabilities are a good proxy, though.

Exactly ! I’d be glad to see such a “generalized Benford law” which might work when data have regularly varying (Pareto type) tails, and a more uniform distribution. Benford is one possible distribution, which works well with the power decay. But alternative distribution should be possible for other underlying distributions, as you mention.

I can’t really understand what the fuss is over Benford’s law personally. At least things like the Pareto law let you focus on smaller subsets and concentrate your efforts.

This aside, is there more power in reviewing the link between the coefficient of variation of the lognormal vs Benford’s law – ie the CoV encompasses all of the lognormal variability in a scale invariant metric?

Fix f(1) in your functional equation, then setting x=1 we get k*f(k) = f(1) for all k, thus f(k) = f(1)/k for all k.