In order to illustrate the problem of chosing a classification model consider some simulated data,
> n = 500 > set.seed(1) > X = rnorm(n) > ma = 10-(X+1.5)^2*2 > mb = -10+(X-1.5)^2*2 > M = cbind(ma,mb) > set.seed(1) > Z = sample(1:2,size=n,replace=TRUE) > Y = ma*(Z==1)+mb*(Z==2)+rnorm(n)*5 > df = data.frame(Z=as.factor(Z),X,Y)
A first strategy is to split the dataset in two parts, a training dataset, and a testing dataset.
> df1 = training = df[1:300,] > df2 = testing = df[301:500,]
- The Holdout Method: Training and Testing Datasets
The two datasets can be visualised below, with the training dataset on top, and the testing dataset below
> plot(df1$X,df1$Y,pch=19,col=c(rgb(1,0,0,.4), + rgb(0,0,1,.4))[df1$Z])