This week, we started to discuss overdispersion when modeling claims frequency. In my previous post, I discussed computations of empirical variances with different exposure. But I did use only one factor to compute classes. Of course, it is possible to use much more factors. For instance, using cartesian products of factors,
> X=as.factor(paste(sinistres$carburant,sinistres$zone,
+ cut(sinistres$ageconducteur,breaks=c(17,24,40,65,101))))
> E=sinistres$exposition
> Y=sinistres$nbre
> vm=vv=ve=rep(NA,length(levels(X)))
> for(i in 1:length(levels(X))){
+ ve[i]=Ei=E[X==levels(X)[i]]
+ Yi=Y[X==levels(X)[i]]
+ vm[i]=meani=weighted.mean(Yi/Ei,Ei) # moyenne
+ vv[i]=variancei=sum((Yi-meani*Ei)^2)/sum(Ei) # variance
+ cat("Class ",levels(X)[i],"average =",meani," variance =",variancei,"\n")
+ }
Class D A (17,24] average = 0.06274415 variance = 0.06174966
Class D A (24,40] average = 0.07271905 variance = 0.07675049
Class D A (40,65] average = 0.05432262 variance = 0.06556844
Class D A (65,101] average = 0.03026999 variance = 0.02960885
Class D B (17,24] average = 0.2383109 variance = 0.2442396
Class D B (24,40] average = 0.06662015 variance = 0.07121064
Class D B (40,65] average = 0.05551854 variance = 0.05543831
Class D B (65,101] average = 0.0556386 variance = 0.0540786
Class D C (17,24] average = 0.1524552 variance = 0.1592623
Class D C (24,40] average = 0.0795852 variance = 0.09091435
Class D C (40,65] average = 0.07554481 variance = 0.08263404
Class D C (65,101] average = 0.06936605 variance = 0.06684982
Class D D (17,24] average = 0.1584052 variance = 0.1552583
Class D D (24,40] average = 0.1079038 variance = 0.121747
Class D D (40,65] average = 0.06989518 variance = 0.07780811
Class D D (65,101] average = 0.0470501 variance = 0.04575461
Class D E (17,24] average = 0.2007164 variance = 0.2647663
Class D E (24,40] average = 0.1121569 variance = 0.1172205
Class D E (40,65] average = 0.106563 variance = 0.1068348
Class D E (65,101] average = 0.1572701 variance = 0.2126338
Class D F (17,24] average = 0.2314815 variance = 0.1616788
Class D F (24,40] average = 0.1690485 variance = 0.1443094
Class D F (40,65] average = 0.08496827 variance = 0.07914423
Class D F (65,101] average = 0.1547769 variance = 0.1442915
Class E A (17,24] average = 0.1275345 variance = 0.1171678
Class E A (24,40] average = 0.04523504 variance = 0.04741449
Class E A (40,65] average = 0.05402834 variance = 0.05427582
Class E A (65,101] average = 0.04176129 variance = 0.04539265
Class E B (17,24] average = 0.1114712 variance = 0.1059153
Class E B (24,40] average = 0.04211314 variance = 0.04068724
Class E B (40,65] average = 0.04987117 variance = 0.05096601
Class E B (65,101] average = 0.03123003 variance = 0.03041192
Class E C (17,24] average = 0.1256302 variance = 0.1310862
Class E C (24,40] average = 0.05118006 variance = 0.05122782
Class E C (40,65] average = 0.05394576 variance = 0.05594004
Class E C (65,101] average = 0.04570239 variance = 0.04422991
Class E D (17,24] average = 0.1777142 variance = 0.1917696
Class E D (24,40] average = 0.06293331 variance = 0.06738658
Class E D (40,65] average = 0.08532688 variance = 0.2378571
Class E D (65,101] average = 0.05442916 variance = 0.05724951
Class E E (17,24] average = 0.1826558 variance = 0.2085505
Class E E (24,40] average = 0.07804062 variance = 0.09637156
Class E E (40,65] average = 0.08191469 variance = 0.08791804
Class E E (65,101] average = 0.1017367 variance = 0.1141004
Class E F (17,24] average = 0 variance = 0
Class E F (24,40] average = 0.07731177 variance = 0.07415932
Class E F (40,65] average = 0.1081142 variance = 0.1074324
Class E F (65,101] average = 0.09071118 variance = 0.1170159
Again, one can plot the variance against the average,
> plot(vm,vv,cex=sqrt(ve),col="grey",pch=19,
+ xlab="Empirical average",ylab="Empirical variance")
> points(vm,vv,cex=sqrt(ve))
> abline(a=0,b=1,lty=2)

An alternative is to use a tree. The tree can be obtained from another variable (the insured had, or had not, a claim, during the period considered) but it should be rather close to the one we would like to model (the number of claims over the period considered). Here, I did use the whole database (with more that 600,000 lines)
> library(tree)
> T=tree((nombre>0)~as.factor(zone)+as.factor(puissance)+
+ as.factor(marque)+as.factor(carburant)+as.factor(region)+
+ agevehicule+ageconducteur,data=baseFREQ,
+ split = "gini",minsize =25000)
The tree is the following
> plot(T)
> text(T)

Now, each knot defines a class, and it is possible to use it to define a class. Which is supposed to be homogeneous.
> X=as.factor(T$where)
> E=sinistres$exposition
> Y=sinistres$nbre
> vm=vv=ve=rep(NA,length(levels(X)))
> for(i in 1:length(levels(X))){
+ ve[i]=Ei=E[X==levels(X)[i]]
+ Yi=Y[X==levels(X)[i]]
+ vm[i]=meani=weighted.mean(Yi/Ei,Ei) # moyenne
+ vv[i]=variancei=sum((Yi-meani*Ei)^2)/sum(Ei) # variance
+ cat("Class ",levels(X)[i],"average =",meani," variance =",variancei,"\n")
+ }
Class 6 average = 0.04010406 variance = 0.04424163
Class 8 average = 0.05191127 variance = 0.05948133
Class 9 average = 0.07442635 variance = 0.08694552
Class 10 average = 0.4143646 variance = 0.4494002
Class 11 average = 0.1917445 variance = 0.1744355
Class 15 average = 0.04754595 variance = 0.05389675
Class 20 average = 0.08129577 variance = 0.0906322
Class 22 average = 0.05813419 variance = 0.07089811
Class 23 average = 0.06123807 variance = 0.07010473
Class 24 average = 0.06707301 variance = 0.07270995
Class 25 average = 0.3164557 variance = 0.2026906
Class 26 average = 0.08705041 variance = 0.108456
Class 27 average = 0.06705214 variance = 0.07174673
Class 30 average = 0.05292652 variance = 0.06127301
Class 31 average = 0.07195285 variance = 0.08620593
Class 32 average = 0.08133722 variance = 0.08960552
Class 34 average = 0.1831559 variance = 0.2010849
Class 39 average = 0.06173885 variance = 0.06573939
Class 41 average = 0.07089419 variance = 0.07102932
Class 44 average = 0.09426152 variance = 0.1032255
Class 47 average = 0.03641669 variance = 0.03869702
Class 49 average = 0.0506601 variance = 0.05089276
Class 50 average = 0.06373107 variance = 0.06536792
Class 51 average = 0.06762947 variance = 0.06926191
Class 56 average = 0.06771764 variance = 0.07122379
Class 57 average = 0.04949142 variance = 0.05086885
Class 58 average = 0.2459016 variance = 0.2451116
Class 59 average = 0.05996851 variance = 0.0615773
Class 61 average = 0.07458053 variance = 0.0818608
Class 63 average = 0.06203737 variance = 0.06249892
Class 64 average = 0.07321618 variance = 0.07603106
Class 66 average = 0.07332127 variance = 0.07262425
Class 68 average = 0.07478147 variance = 0.07884597
Class 70 average = 0.06566728 variance = 0.06749411
Class 71 average = 0.09159605 variance = 0.09434413
Class 75 average = 0.03228927 variance = 0.03403198
Class 76 average = 0.04630848 variance = 0.04861813
Class 78 average = 0.05342351 variance = 0.05626653
Class 79 average = 0.05778622 variance = 0.05987139
Class 80 average = 0.0374993 variance = 0.0385351
Class 83 average = 0.06721729 variance = 0.07295168
Class 86 average = 0.09888492 variance = 0.1131409
Class 87 average = 0.1019186 variance = 0.2051122
Class 88 average = 0.05281703 variance = 0.0635244
Class 91 average = 0.08332136 variance = 0.09067632
Class 96 average = 0.07682093 variance = 0.08144446
Class 97 average = 0.0792268 variance = 0.08092019
Class 99 average = 0.1019089 variance = 0.1072126
Class 100 average = 0.1018262 variance = 0.1081117
Class 101 average = 0.1106647 variance = 0.1151819
Class 103 average = 0.08147644 variance = 0.08411685
Class 104 average = 0.06456508 variance = 0.06801061
Class 107 average = 0.1197225 variance = 0.1250056
Class 108 average = 0.0924619 variance = 0.09845582
Class 109 average = 0.1198932 variance = 0.1209162
Here, when ploting the empirical variance (per knot) against the empirial average of claims, we get

Here, we can identify classes where remaining heterogeneity.