Heuristics on bias and variance for kernel density estimators

Consider the simple case of a moving histogram (which is a very simple kernel). The idea is to recall that


is the slope close to point .

Then we use the empirical cumulative density to approximate the slope, i.e.

which can also be writen

Consider now the density seen as a random variable

where the‘s are i.i.d. where , with

Thus, observe that , but that’s not what we’re looking for… From Taylor’s expansion,


where the bias comes from the approximation of the density by some string. About the variance,

thus, since ,


We can observe that

is decreasing as , while the variance is increasing as . This is the standard bias-variance tradeoff in statistics.

One thought on “Heuristics on bias and variance for kernel density estimators”

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.