19 Feb 2016

Integrating custom models into the caret framework

Well, this is going to be a pretty short post. Max Kuhn has published instructions online as to how you go about integrating custom models with caret. I recently wrote my nnePtR package, which I detailed in this post. So I here is where I give it a go, and unlock the magic caret-ty goodness for my model!

setting up nneptr for use with caret

Following Max's instructions, we define the following list, so that we can use our custom model with caret::train()

#
# start by naming my method to pass to train
#
nnetP <- list(type = Classification,
              library = nnePtR,
              loop = NULL)
#
# define the tuning parameters
#
prm <- data.frame(parameter = c(nLayers, lambda),
                  class = rep(numeric, 2),
                  label = c(Hidden Layers, Penalty))
#
# append them to the list
#
nnetP$parameters <- prm
#
# define the default training grid. some models can do 
# a random search, but I wont implement that
#
nnetPGrid <- function(x, y, len = NULL, search = grid) {
  if(search == grid) {
    out <- expand.grid(nLayers = seq(1, 5, 2),
                       lambda = c(0.01, 0.03))
  } else {
    stop('random search not yet implemented')
  }
  out
}
#
# append to list
#
nnetP$grid <- nnetPGrid
#
# define the fitting function. Here, it is the 
# nnePtR constructor function nnetBuild()
#
nnetPFit <- function(x, y, wts, param, lev, last, weights, classProbs, ...) {
  library(nnePtR)
  nnetBuild(train_input = x,
                    train_outcome = y,
                    nLayers = param$nLayers,
                    lambda = param$lambda,
                    ...)
}
#
# append to list
#
nnetP$fit <- nnetPFit
#
# define the levels of the outcome.
# they are held in the levels slot of objects of
# class nnePtR
#
nnetP$levels <- function(x) x@levels
#
# define the classification prediction with the
# predict generic
#
nnetPPred <- function(modelFit, newdata, preProc = NULL, submodels = NULL) {
  predict(modelFit, newdata)
}
#
# append to list
#
nnetP$predict <- nnetPPred
#
# define the class probability with the
# predict generic
#
nnetPProb <- function(modelFit, newdata, preProc = NULL, submodels = NULL) {
  predict(modelFit, newdata, type = prob)
}
#
# append to list
#
nnetP$prob <- nnetPProb
#
# define the sort function, i.e. how the tuning parameters
# are ordered in case similar performance obtained
#
nnetPSort <- function (x) x[order(x$nLayers, -x$lambda), ]
#
# append to list
#
nnetP$sort <- nnetPSort

And believe it or not (I was a little suprised), that is it! we are ready to go!

testing out on the iris data set

Right, lets put this to the test. The iris data set seems to be one of my favorites (I seem to always use it for examples!), so lets train the model to this data set. First, load up the caret package, and if you want to speed up the training a bit, load the doMC package for parallel training.

The code snippet below details how I specify the training options (repeated 10-fold CV) and define the training grid of parameters (else the default parameters will be used).

I define an object of class train called nnetTune. I have access to the preprocessing options, so I specify that I want to center and scale the predictive inputs. I can pass other arguments to the nnePtR cpnstructor function, so I specify that I want 20 units per hidden layer in my network. And then, we pretty much just hit go!

# load caret and doMC library
library(caret)
library(doMC)
#
# register cores for parallel processing
#
registerDoMC(4)
#
# train control options. want repeated 10-fold CV
#
fitControl <- trainControl(method = repeatedcv,
                           number = 10,
                           repeats = 5)
#
# define grid of parameter values
#
nnetGrid <- expand.grid(lambda = c(0.1, 0.3),
                        nLayers = seq(1:5))
#
# train using caret::train()
# preprocess by normalising inputs
# pass nUnits directly to nnePtR constructor function
#
set.seed(825)
nnetTune <- train(x = iris[, 1:4],
                  y = iris[, 5],
                  method = nnetP,
                  trControl = fitControl,
                  tuneGrid = nnetGrid,
                  preProcess = c(center, scale),
                  nUnits = 20)
#
# we have access to plot generic for object of class train
#
plot(nnetTune)
#
# and the predict generic!
#
predict(nnetTune, newdata = iris[, 1:4])
predict(nnetTune, newdata = iris[, 1:4], type = prob)

We get back an object of class train, so we get the nicely formatted summary printed to screen that comes with it, as well as the plot and predict generics

For example, lets plot the resamples' accuracy over the different training parameters:

caret_resample_philip_goddard"

We see peformance roughly constant until number of hidden layers exceeds 3- then it drops off sharply to a pretty much uninformative model (remember we have three balanced classes, so random guessing should get accuracy of 0.33).

Final thoughts

Well, that was short and sweet! Max Kuhn has left the caret framework accessable for custom models, and has documented the process thoroughly. I can now access all the benefits of the caret model training framework (such as preprocessing and cross validation) with my nnePtR model. This is great- saves me a job of having to implement this all myself. Thanks Max!

TL;DR I had a quick go at adding a custom model into the caret framework. It was painless!