ROC curves and classification

September 30, 2013
By

To get back to a question asked after the last course (still on non-life insurance), I will spend some time to discuss ROC curve construction, and interpretation. Consider the dataset we’ve been using last week,

> db = read.table("http://freakonometrics.free.fr/db.txt",header=TRUE,sep=";")
> attach(db)

The first step is to get a model. For instance, a logistic regression, where some factors were merged together,

> X3bis=rep(NA,length(X3))
> X3bis[X3%in%c("A","C","D")]="ACD"
> X3bis[X3%in%c("B","E")]="BE"
> db$X3bis=as.factor(X3bis)
> reg=glm(Y~X1+X2+X3bis,family=binomial,data=db)

From this model, we can predict a probability, not a  variable,

> S=predict(reg,type="response")

Let http://latex.codecogs.com/gif.latex?\widehat{S} denote this variable (actually, we can use the score, or the predicted probability, it will not change the construction of our ROC curve). What if we really want to predict a  variable. As we usually do in decision theory. The idea is to consider a threshold http://perso.univ-rennes1.fr/arthur.charpentier/latex/ROC-04.png, so that

  • if http://perso.univ-rennes1.fr/arthur.charpentier/latex/ROC-05.png, then  http://perso.univ-rennes1.fr/arthur.charpentier/latex/ROC-02.png will be http://latex.codecogs.com/gif.latex?1, or “positive” (using a standard terminology)
  • si http://perso.univ-rennes1.fr/arthur.charpentier/latex/ROC-06.png, then  http://perso.univ-rennes1.fr/arthur.charpentier/latex/ROC-02.png will be http://latex.codecogs.com/gif.latex?0, or “negative

Then we derive a contingency table, or a confusion matrix

     observed value http://perso.univ-rennes1.fr/arthur.charpentier/latex/ROC-01.png
predicted
value
http://perso.univ-rennes1.fr/arthur.charpentier/latex/ROC-02.png
“positive“ “négative“
“positive“ TP FP
“négative“ FN TN

where TP are the so-called true positive, TN  the true negative, FP are the false positive (or type I error) and FN are the false negative (type II errors). We can get that contingency table for a given threshold http://perso.univ-rennes1.fr/arthur.charpentier/latex/ROC-04.png

> roc.curve=function(s,print=FALSE){
+ Ps=(S>s)*1
+ FP=sum((Ps==1)*(Y==0))/sum(Y==0)
+ TP=sum((Ps==1)*(Y==1))/sum(Y==1)
+ if(print==TRUE){
+ print(table(Observed=Y,Predicted=Ps))
+ }
+ vect=c(FP,TP)
+ names(vect)=c("FPR","TPR")
+ return(vect)
+ }
> threshold = 0.5
> roc.curve(threshold,print=TRUE)
        Predicted
Observed   0   1
       0   5 231
       1  19 745
      FPR       TPR 
0.9788136 0.9751309

Here, we also compute the false positive rates, and the true positive rates,

  • TPR = TP / P = TP / (TP + FN) also called sentivity, defined as the rate of true positive: probability to be predicted positve, given that someone is positive (true positive rate)
  • FPR = FP / N = FP / (FP + TN) is the rate of false positive: probability to be predicted positve, given that someone is negative (false positive rate)

The ROC curve is then obtained using severall values for the threshold. For convenience, define

> ROC.curve=Vectorize(roc.curve)

First, we can plot http://latex.codecogs.com/gif.latex?(\widehat{S}_i,Y_i) (a standard predicted versus observed graph), and visualize true and false positive and negative, using simple colors

> I=(((S>threshold)&(Y==0))|((S<=threshold)&(Y==1)))
> plot(S,Y,col=c("red","blue")[I+1],pch=19,cex=.7,,xlab="",ylab="")
> abline(v=threshold,col="gray")

And for the ROC curve, simply use

> M.ROC=ROC.curve(seq(0,1,by=.01))
> plot(M.ROC[1,],M.ROC[2,],col="grey",lwd=2,type="l")

This is the ROC curve. Now, to see why it can be interesting, we need a second model. Consider for instance a classification tree

> library(tree)
> ctr <- tree(Y~X1+X2+X3bis,data=db)
> plot(ctr)
> text(ctr)

To plot the ROC curve, we just need to use the prediction obtained using this second model,

> S=predict(ctr)

All the code described above can be used. Again, we can plot http://latex.codecogs.com/gif.latex?(\widehat{S}_i,Y_i) (observe that we have 5 possible values for http://latex.codecogs.com/gif.latex?\widehat{S}_i, which makes sense since we do have 5 leaves on our tree). Then, we can plot the ROC curve,

An interesting idea can be to plot the two ROC curves on the same graph, in order to compare the two models

> plot(M.ROC[1,],M.ROC[2,],type="l")
> lines(M.ROC.tree[1,],M.ROC.tree[2,],type="l",col="grey",lwd=2)

The most difficult part is to get a proper interpretation. The tree is not predicting well in the lower part of the curve. This concerns people with a very high predicted probability. If our interest is more on those with a probability lower than 90%, then, we have to admit that the tree is doing a good job, since the ROC curve is always higher, comparer with the logistic regression.


Print This Post Print This Post

Tags: , , ,

Posted in: Course, ACT2040, 2013, Course, ACT2040, Course, ACT2040, 2013, Automne, Course, Statistics, Datamining, Statistics, GLM

23 Responses to “ ROC curves and classification ”

  1. Phillip Ludwig on 16/12/2013 at 9:23 PM

    Thank-you Arthur for your post. It is clear and easy to follow. I am relatively new to R & have chosen it as one of my dissertation tools. It requires that one knows a lot more than point-n-click (e.g. SPSS). However, it is definitely more exacting and thereby has less forgiving of not thoroughly befriending one’s dataset. I have racked my brains and tried everything I could for the last two days to make this code work for my project. I keep getting the following error:
    “Error in table(Observed = TX, Predicted = Ps) :
    all arguments must have the same length
    In addition: Warning messages:
    1: In (Ps == 1) * (TX == 0) :
    longer object length is not a multiple of shorter object length
    2: In (Ps == 1) * (TX == 1) :
    longer object length is not a multiple of shorter object length”.
    This (of course) is not tremendously helpful to me. I cannot figure out what is different in my model that does not allow your elegant programme to work. HELP please, Phillip

    • Arthur Charpentier on 16/12/2013 at 10:34 PM

      I do not see why you get those TX variables… indeed, they were not defined here…
      please, see also http://freakonometrics.hypotheses.org/9307 (posted just after that post actually)

      • Phillip on 17/12/2013 at 4:41 PM

        Hi Arthur,

        Comme vous devinez, c’etais les cas incomplets. Merci mille fois.

        Phillip

  2. Bailei on 13/11/2013 at 9:45 PM

    Hello!Thanks a lot for sharing.
    I have a question.
    M.ROC=ROC.curve(seq(0,1,by=.01))
    the parameters of ROC.curve() is “threshold”? or what is the parameters of ROC.curve()?

  3. » Logistic regression MarkR on 23/10/2013 at 11:06 PM

    [...] ROC curves and classification [...]

  4. Raja Fan on 02/10/2013 at 10:07 PM

    Very clear and excellent post. Thanks a lot for sharing.

  5. Alistair Walsh on 02/10/2013 at 1:48 PM

    agreed. It didn’t take me too long to work it out.

  6. Alistair Walsh on 02/10/2013 at 12:40 AM

    After running
    plot(M.ROC[1,],M.ROC[2,],type=”l”)
    lines(M.ROC.tree[1,],M.ROC.tree[2,],type=”l”,col=”grey”,lwd=2)

    I’m getting an error:
    Error in lines(M.ROC.tree[1, ], M.ROC.tree[2, ], type = “l”, col = “grey”, :
    object ‘M.ROC.tree’ not found

    Have I missed something?

    • Arthur Charpentier on 02/10/2013 at 7:23 AM

      yes, I did not type all the code, since it is exactly the same as before… this is a code for my students (mainly), and after some experience, I usually upload incomplete code. It takes 2 minutes to complete it, but I want my student to work a little bit, and to aviod copy and paste of the code.

  7. Frank Harrell on 30/09/2013 at 8:59 PM

    Choosing a cutpoint in this manner is at odds with decision theory. Decision theory tells us that a cutpoint cannot be chosen without a loss/utility/cost function being specified.

    • Arthur Charpentier on 30/09/2013 at 9:27 PM

      Indeed. But this was not my point in this post. Actually, I mention decision theory in my course to mention that you can not get a decision ‘process’ that will give you no-error. You can easily get a decision rule that will give no False Positive, and you can get another decision rule that will give no False Negative. What I explain in my course is that any decision rule is somehow a trade-off between the two types of errors, and indeed, it is related to some loss/utility/cost function. Based on a rule, you will get a cutoff point, but here, we would like to see what happens for all possible cut-offs. I guess your point can be related to my latest paragraph, where I mention that comparing ROC curve is not that simple, and again, it does where your focus is, which is the same as ‘it depends what your loss function is’.

      • Frank Harrell on 30/09/2013 at 9:36 PM

        I think this example will lull many readers to use cutpoints where that use will be inconsistent about what we know about optimal decisions. Decision theory shows that there is no single optimal cutpoint except in the unusual case that every subject has the same utility function and you know the function. ROC curves are only useful for one-time group decision making (“one size fits all”).

        • Arthur Charpentier on 30/09/2013 at 9:55 PM

          Perhaps I was not clear… my point is not ‘ROC curves are great’ or ‘ROC curves are useless’. My point was more ‘ROC curves exist and a lot of people use them’. Some students came back to me last week, asking ‘How do we interpret them?’. And the way I see interpretation is ‘if you want to discuss about a curve, first, we have to understand how that curve what plotted’. So I wrote a post to explain how those curves where plotted. ROC curves are based on the idea of using several cutoff points, and for each cutoff point, I see a Decision Theory problem, like ‘that point can be obtained for someone with a very specific loss or utility curve’. My course is not a course on theoretical statistics, but more a statistical toolbox for actuaries. And since the ROC curve is popular (for good reasons or not), I want my students to understand how this curve is obtained. And what could be the possible interpretations.

          • Frank Harrell on 01/10/2013 at 8:13 AM

            The fact that a lot of people are drawing ROC curves does not mean that they are useful for the intended purposes. ROC curves play absolutely no role in optimal Bayes decisions, which uses full conditioning and a loss function. Conditioning on X=x instead of X>x is consistent with using the available data. Incomplete conditioning such as X>x causes many artifacts and misinterpretations.

        • Byron on 01/10/2013 at 8:43 AM

          ROC curves exist and are used. Quite often they are used regardless of whether they are decision theoretically optimal due to ease of use, time constraints, and ease of explanation to the non-stat inclined.

          If you have an alternative that you prefer please create a post as I would love to read more.

          Thank you Arthur for the clear, easy to understand descriptions.

          • Frank Harrell on 01/10/2013 at 1:51 PM

            Again, the fact that they are used does not mean they are relevant or well understood. Understanding probabilities and checking their absolute accuracy (e.g., with nonparametric calibration curves) is more important in my opinion.

  8. Gu Mi on 30/09/2013 at 7:40 PM

    Thanks for the great post! I have two questions:

    1. For the code: abline(v=seuil,col=”gray”), can I know where you defined “seuil”?
    2. How you made the interactive side-by-side plot? The codes produce only static plots.

    Thank you so much :)

    • Arthur Charpentier on 30/09/2013 at 7:45 PM

      sorry, ‘seuil’ is the French for ‘threshold’. I did type in French, and forgot to translate that part when I posted the code on the blog. I did change it in the text. And indeed, the code produces statics plot. You can use the animate package and a loop on s (here my loop was based on values from 100% to 40$, with -1% for each picture).

      • Gu Mi on 01/10/2013 at 12:04 AM

        Thank you so much for the clarifications! I tried to use the animation package but failed to find the correct functions… Is it possible that you post the codes for the animation part? I am really interested in how it works :) It is totally find if you prefer to make it private. Thanks in advance!

  9. Hywel Jones on 30/09/2013 at 4:34 PM

    ‘Sensitivity’, not ‘sensibility’!

    • Arthur Charpentier on 30/09/2013 at 4:43 PM

      ooops, thanks ! such a shame… Today was a busy and stressful day (promise, I will get back to the story which started a few days ago) .

      • Cécile Sauder on 01/10/2013 at 2:41 PM

        Don’t worry Arthur, it’s not a shame, when you’re french you don’t notice the error ;)

Leave a Reply

Categories

Students

Keep calm…

Keep calm…

Research blogs