1 Introduction

The goal of this document is to describe the infra-structure that is available in performanceEstimation¹ to estimate the performance of different approaches to predictive tasks. The main goal of this package is to provide a general infra-structure that can be used to estimate the performance using several predictive performance metrics of any modelling approach to different predictive tasks, with a minimal effort from the user. The package provides this type of facilities for classification, regression and time series tasks. There is no limitation on the type of approaches to these tasks for which you can estimate the performance - the user just needs to provide a workflow function (following some interface rules) that implements the approach for which the predictive performance is to be estimated. The package includes some standard workflow functions that implement the typical learn+test approach that most users will be interested in. This means that if you just want to estimate the performance of some variants of a method already implemented in R (e.g. an SVM), on some particular tasks, you will be able to use these standard workflow functions and thus your required input will be limited to the minimum. The package also provides a series of predictive performance metrics for different tasks. Still, you are not limited to these metrics and can use any metric as long as it exists in R or you provide a function calculating it. Finally, the package also include several standard data pre-processing and predictions post-processing steps that again can be incorporated into the workflows being evaluated.

This infra-structure implements different methods for estimating the predictive performance. Namely, you can select among: (i) cross validation, (ii) holdout and random sub-sampling, (iii) leave one out cross validation, (iv) bootstrap (\(\epsilon_0\) and .631) and also (v) Monte-Carlo experiments for time series forecasting tasks. For each of these tasks different options are implemented (e.g. use of stratified sampling).

Most of the times experimental methodologies for performance estimation are iterative processes that repeat the modeling task several times using different train+test samples with the goal of improving the accuracy of the estimates. The estimates are the result of aggregating the scores obtained on each of the repetitions. For each of these repetitions different training and testing samples are generated and the process being evaluated is “asked” to: (i) obtain the predictive model using the training data, and then (ii) use this model to obtain predictions for the respective test sample. These predictions can then be used to calculate the scores of the performance metrics being estimated. This means that there is a workflow that starts with a predictive task for which training and testing samples are given, and that it should produce as result the predictions of the workflow for the given test sample. There are far too many possible approaches and sub-steps for the implementation of this workflow. To ensure full generality of the infra-structure, we allow the user to provide a function that implements this workflow for each of the predictive approaches she/he wishes to compare and/or evaluate. This function can be parameterizable in the sense that there may be variants of the workflow that the user wishes to evaluate and/or compare. Still, the goal of this workflow functions is very clear: (i) receive as input a predictive task for which training and test samples are given, as well as any eventual workflow specific parameters; and (ii) produce as result a set of predictions for the given test set. These predictions will then be used to obtain the scores of the predictive metrics for which the user is interested in obtaining reliable estimates.

The infra-structure we describe here provides means for the user to indicate: (i) a set of predictive tasks with the respective data sets; (ii) a set of workflows and respective variants; and (iii) the information on the estimation task. The infra-structure then takes care of all the process of experimentally estimating the predictive performance of the different approaches on the tasks, producing as result an object that can be explored in different ways to obtain the results of the estimation process. The infra-structure also provides several utility functions to explore these results objects, for instance to obtain summaries of the estimation process both in textual format as well as visually. Moreover, it also provides functions that carry out statistical significance tests based on the outcome of the experiments.

Finally, the infra-structure provides utility functions implementing frequently used workflows for common modelling techniques, several data pre-processing steps and prediction post-processing operations, as well as functions that facilitate the automatic generation of variants of workflows by specifying sets of parameters that the user wishes to consider in the comparisons.

2 Package Installation

The package can be installed as any other R package that is available at the R central repository (CRAN), i.e. by simply executing the code:

install.packages("performanceEstimation")

This is the recommended form of installing the package an we assume that you start by doing this before you proceed to try the examples described in this document. This will install the current stable version of the package that at the time of writing of this document is version 1.1.0.

If you wish to try some eventual new developments of the package that are still under testing and have not yet been released to the general public (thus are subject to bugs), then you may install the development version of the package that is available at the package GitHub Web page:

https://github.com/ltorgo/performanceEstimation

In order to install this development version (again not recommended unless you have a good reason for it), you may issue the following commands in R:

library(devtools)  # You need to install this package before!
install_github("ltorgo/performanceEstimation",ref="develop")

The GitHub Web page referred above is also the right place for you to report any issues you have with the package or help in its development.

3 A Simple Illustrative Example

Let us assume we are interested in estimating the predictive performance of several variants of an SVM on the Iris classification problem. More specifically, we want to obtain a reliable estimate of the error rate of these variants using 10-fold cross validation. The following code illustrates how these estimates could be obtained with our proposed infra-structure.

library(performanceEstimation)  # Loading our infra-structure
library(e1071)                  # A package containing SVMs

data(iris)                      # The data set we are going to use

res <- performanceEstimation(
         PredTask(Species ~ .,iris),
         Workflow("standardWF",learner="svm"),
         EstimationTask(metrics="err",method=CV())
         )  

## 
## 
## ##### PERFORMANCE ESTIMATION USING  CROSS VALIDATION  #####
## 
## ** PREDICTIVE TASK :: iris.Species
## 
## ++ MODEL/WORKFLOW :: svm 
## Task for estimating  err  using
##  1 x 10 - Fold Cross Validation
##   Run with seed =  1234 
## Iteration :**********

This simple example illustrates several key concepts of our infra-structure. First of all, we have the main function - performanceEstimation(), which is used to carry out the estimation process. It has 3 main arguments: (i) a vector of predictive tasks (in the example a single one); (ii) a vector of workflows (in the example a single one); and (iii) the specification of the estimation task (essentially the metrics to be estimated and the estimation methodology). The package defines three classes of objects for storing the information related with these three concepts: predictive tasks (S4 class PredTask); workflows (S4 class Workflow); and estimation tasks (S4 class EstimationTask).

PredTask objects can be created by providing information on the formula defining the task, the source data set (an R data frame or the name of such object), and an optional ID (a string) to give to the task.

Workflow objects include information on the name of the function implementing the workflow and any number of parameters to be passed to this function when it will be called with different train and test sets.

EstimationTask objects provide information on the metrics for which we want a reliable estimate and also the methodology to be used to obtain these estimation. In case of metrics not defined with the performanceEstimation we also need to supply the name of the function that can be used to calculate these specific metrics.

In the above simple example we are using the data frame iris to create a task consisting of forecasting the variable Species using all remaining variables. We are solving this classification task by using the function standardWF with the parameter learner set to svm. This workflow function is already provided by the package and essentially can be used for the most frequent situations where a user just wants to apply and existing learning method to solve some task. This avoids the need for the user to have to write the functions solving the task. The standardWF function simply applies the modeling function indicated through the parameter learner (in this case the function svm defined in package e1071 (Meyer et al. 2012)) to the training set, and uses the resulting model to obtain the predictions for the test set. As we will see later the standardWF function also implements a series of common data pre-processing steps (e.g. filling in unknown values), and several common prediction post-processing steps (e.g. applying some transformation to the predicted values). Finally, the above example specifies the estimation task as using 10-fold Cross Validation to obtain estimates of the error rate.

The result of the call to performanceEstimation() is an S4 object of the class ComparisonResults. These objects tipically are not directly explored by the end-user so we ommit their details here². There are several utility functions that allow the users to explore the results of the experimental comparisons, as shown by the next few illustrative examples.

summary(res)

## 
## == Summary of a  Cross Validation Performance Estimation Experiment ==
## 
## Task for estimating  err  using
##  1 x 10 - Fold Cross Validation
##   Run with seed =  1234 
## 
## * Predictive Tasks ::  iris.Species
## * Workflows  ::  svm 
## 
## -> Task:  iris.Species
##   *Workflow: svm 
##                err
## avg     0.03333333
## std     0.04714045
## med     0.00000000
## iqr     0.06666667
## min     0.00000000
## max     0.13333333
## invalid 0.00000000

The generic function summary allows us to obtain the estimated scores for each compared approach on each predictive task. For each performance metric (in this case only the error rate), the function shows several descriptive statistics of the performance of the workflow in the different iterations of the estimation process. Moreover, information is also given on eventual failures on some of the iterations.

The generic function plot can be used to obtain a graphical display of the distribution of performance metrics across the different iterations of the estimation process using box-plots, as show in the following figure:

The distribution of the error rate on the 10 folds. Red dots show the performance on each individual iteration of the estimation process.

The above example is just a simple illustration of the key concepts of the performanceEstimation. Most of the times you will use the package for more complex experiments, usually involving several tasks, several workflows or workflow variants, and eventually estimating several performance metrics. Before, we provide illustrations for these more complex setups we describe in more detail the key concepts of the performanceEstimation.

4 Predictive Tasks

Predictive tasks are data analysis problems where we want to obtain a model of an unknown function \(Y=f(X_1, X_2, \cdots, X_p)\) that relates a target variable \(Y\) with a set of \(p\) predictors \(X_1, X_2, \cdots, X_p\). The model is usually obtained using a sample of \(n\) observations of the mapping of the unknown function, \(D=\{\langle \mathbf{x}_i, Y_i\rangle\}_{i=1}^n\), where \(\mathbf{x}_i\) is a vector with the \(p\) predictors values and \(Y_i\) the respective target variable value. These data sets in R are usually stored in data frames, and formula objects are used to specify the form of the functional dependency that we are trying to model, i.e. which is the target variable and the predictors.

Objects of class PredTask encapsulate the information of a predictive task, i.e. the functional form and the data required for solving it. For convenience they also allow the user to assign a name to each task. These S4 objects can be created using the construtor function PredTask(), as seen in the following example:

data(iris)
PredTask(Species ~ .,iris)

## Prediction Task Object:
##  Task Name         :: iris.Species 
##  Task Type         :: classification 
##  Target Feature    :: Species 
##  Formula           :: Species ~ .
##  Task Data Source  :: iris

We should remark that, by default and for memory economy particularly if you are using large data sets, the objects of this class do not copy internally the data of the provided data frame. This means that they assume that the data frame provided in the second argument will still exist when the estimation experiments will be executed. If you want the data to be copied into the PredTask object then you should call the constructor as follows:

data(iris)
PredTask(Species ~ .,iris,copy=TRUE)

## Prediction Task Object:
##  Task Name         :: iris.Species 
##  Task Type         :: classification 
##  Target Feature    :: Species 
##  Formula           :: Species ~ .
##  Task Data Source  :: internal  150x5 data frame.

Note the difference on the task data source information to indicate that the data set is stored internally (i.e. a copy was made from the provided object).

5 Workflows and Workflow Variants

Estimation methodologies work most of the times by re-sampling the available data set \(D\) in order to create different train and test samples from \(D\) (an exception being a single repetition of hold-out). The goal is to estimate the predictive performance of a proposed workflow to solve the task, by using these different samples to increase our confidence on the estimates. This workflow consists on the process of obtaining a model from a given training sample and then use it to obtain predictions for the given test set. This process can include several steps, e.g. specific data pre-processing steps, and may use any modelling approach, eventually developed by the user.

We distinguish two types of workflows: (i) generic, and (ii) user-defined. The first are probably the most common setups where the user simply wants to apply methods that already exist in R to some task. The second covers situations where the user has developed her/his own specific approach to solve a task and wants to apply it to some concrete problems. The package covers both situations as we will see.

Independently of the workflows being generic or user-defined, they will typically have parameters that we can tune to try to improve the results. Frequently, we want to estimate and compare the performance of these parameter variants over some set of tasks. Our package also facilitates these setups by providing a function to easily specify these alternatives.

library(performanceEstimation)
library(e1071)
data(Boston,package="MASS")                    

exp <- performanceEstimation(
         PredTask(medv ~ .,Boston),
         workflowVariants(wf="standardWF",learner="svm",
                          learner.pars=list(cost=1:5,gamma=c(0.1,0.05,0.01))),
         EstimationTask(metrics="mae",method=CV())
         )  

library(performanceEstimation)
library(DMwR)
data(iris)                    

res <- performanceEstimation(
         PredTask(Species ~ .,iris),
         workflowVariants(learner="rpartXse",
                          learner.pars=list(se=c(0,0.5,1)),
                          predictor.pars=list(type="class")),
         EstimationTask(metrics="acc",method=CV(nReps=2,nFolds=5))
         )  

data(Boston,package='MASS')
library(DMwR)
library(e1071)
library(randomForest)
bostonRes <- performanceEstimation(
  PredTask(medv ~ .,Boston),
  workflowVariants(learner=c('rpartXse','svm','randomForest')),
  EstimationTask(metrics="mse",method=CV())
  )

data(Boston,package='MASS')
library(DMwR)
library(e1071)
library(randomForest)
bostonRes <- performanceEstimation(
  PredTask(medv ~ .,Boston),
  c(workflowVariants(learner='rpartXse',
                     learner.pars=list(se=c(0,1))),
    workflowVariants(learner='svm',
                     learner.pars=list(cost=c(1,5),gamma=c(0.01,0.1))),
    workflowVariants(learner='randomForest',
                     learner.pars=list(ntree=c(500,1000)))),
  EstimationTask(metrics="mse",method=CV())
  )

library(performanceEstimation)
data(algae,package="DMwR")

res <- performanceEstimation(
         PredTask(a1 ~ .,algae[,1:12],"AlgaA1"),
         Workflow(learner="lm",pre=c("centralImp","scale")),
         EstimationTask(metrics="mae",method=CV())
         )

## 
## 
## ##### PERFORMANCE ESTIMATION USING  CROSS VALIDATION  #####
## 
## ** PREDICTIVE TASK :: AlgaA1
## 
## ++ MODEL/WORKFLOW :: lm 
## Task for estimating  mae  using
##  1 x 10 - Fold Cross Validation
##   Run with seed =  1234 
## Iteration :**********

library(performanceEstimation)
data(algae,package="DMwR")

res <- performanceEstimation(
         PredTask(a1 ~ .,algae[,1:12],"AlgaA1"),
         c(Workflow(wfID="lm",
                    learner="lm",
                    pre=c("centralImp","scale")),
           Workflow(wfID="lmOnlyPos",
                    learner="lm",
                    pre=c("centralImp","scale"),
                    post=c("onlyPos"))),
         EstimationTask(metrics="mae",method=CV())
         ) 

## 
## 
## ##### PERFORMANCE ESTIMATION USING  CROSS VALIDATION  #####
## 
## ** PREDICTIVE TASK :: AlgaA1
## 
## ++ MODEL/WORKFLOW :: lm 
## Task for estimating  mae  using
##  1 x 10 - Fold Cross Validation
##   Run with seed =  1234 
## Iteration :**********
## 
## 
## ++ MODEL/WORKFLOW :: lmOnlyPos 
## Task for estimating  mae  using
##  1 x 10 - Fold Cross Validation
##   Run with seed =  1234 
## Iteration :**********

summary(res)

## 
## == Summary of a  Cross Validation Performance Estimation Experiment ==
## 
## Task for estimating  mae  using
##  1 x 10 - Fold Cross Validation
##   Run with seed =  1234 
## 
## * Predictive Tasks ::  AlgaA1
## * Workflows  ::  lm, lmOnlyPos 
## 
## -> Task:  AlgaA1
##   *Workflow: lm 
##               mae
## avg     14.308536
## std      1.857755
## med     14.340022
## iqr      2.447854
## min     11.652404
## max     17.451525
## invalid  0.000000
## 
##   *Workflow: lmOnlyPos 
##               mae
## avg     13.116271
## std      1.690161
## med     13.173095
## iqr      2.273219
## min      9.955684
## max     15.302897
## invalid  0.000000

library(performanceEstimation)
library(quantmod)
library(randomForest)
GSPC <- getSymbols('^GSPC',from='2008-01-01',to='2012-12-31',auto.assign = FALSE)
data.model <- specifyModel(
  Next(100*Delt(Ad(GSPC))) ~ Delt(Ad(GSPC),k=1:10))
data <- as.data.frame(modelData(data.model))
colnames(data)[1] <- 'PercVarClose'
spExp <- performanceEstimation(
  PredTask(PercVarClose ~ .,data,'SP500_2012'),
  c(Workflow(wf='standardWF',wfID="standRF",
             learner='randomForest',
             learner.pars=list(ntree=500)),
    Workflow(wf='timeseriesWF',wfID="slideRF",
             learner='randomForest',
             learner.pars=list(ntree=500),
             type="slide",
             relearn.step=30),
    Workflow(wf='timeseriesWF',wfID="growRF",
             learner='randomForest',
             learner.pars=list(ntree=500),
             type="grow",
             relearn.step=30)
   ),
  EstimationTask(metrics=c("mse","theil"),
                 method=MonteCarlo(nReps=5,szTrain=0.5,szTest=0.25))
  )

myWorkFlow <- function(form,train,test,...) {
  require(mySpecialPackage,quietly=TRUE)
  ## cary out some data pre-processing
  myTrain <- mySpecificPreProcessingSteps(train)
  ## now obtain the model
  myModel <- myModelingTechnique(form,myTrain,...)
  ## obtain the predictions
  preds <- predict(myModel,test)
  ## cary out some predictions post-processing
  newPreds <- mySpecificPostProcessingSteps(form,train,test,preds)
  names(newPreds) <- rownames(test)
  ## finally produce the list containing the output of the workflow
  res <- list(trues=responseValues(form,test),preds=newPreds)
  return(res)
}

workflowVariants('myWorkFlow',x=c(0,3,5,7),y=c(TRUE,FALSE))

RLensemble <- function(f, tr, ts, weightRT=0.5, step=FALSE, ..., .models=FALSE) {
  require(DMwR,quietly=TRUE)
  ## Getting the column id of the target variable
  tgtCol <- which(colnames(tr) == as.character(f[[2]]))
  ## filling in NAs using knnImputation
  noNAsTR <- tr
  noNAsTS <- ts
  noNAsTR[,-tgtCol] <- knnImputation(tr[,-tgtCol])
  noNAsTS[,-tgtCol] <- knnImputation(ts[,-tgtCol],distData=tr[,-tgtCol])
  r <- rpartXse(f,tr,...)
  l <- lm(f,noNAsTR)
  if (step) l <- step(l,trace=0)
  pr <- predict(r,ts)
  pl <- predict(l,noNAsTS) 
  ps <- weightRT*pr+(1-weightRT)*pl
  names(ps) <- rownames(ts)
  res <- list(trues=responseValues(f,ts),preds=ps)
  if (.models) res <- c(res,list(linearModel=l,tree=r))
  res
}

data(algae,package='DMwR')
expRes <- performanceEstimation(
    PredTask(a1 ~ .,algae[,1:12],'alga1'),
    workflowVariants('RLensemble',
                     se=c(0,1),step=c(TRUE,FALSE),weightRT=c(0.4,0.5,0.6)),
    EstimationTask("mse",method=CV()))

6 Estimation Tasks

The third argument of the main function performanceEstimation defines the estimation task we want to carry out. Its value is an S4 object of class EstimationTask that can be created through the constructor function with the same name. The main arguments of this constructor are metricsand method. The first is a vector with names of metrics for which we want reliable estimates, while the second is the method to be used to obtain these estimates. Other arguments of the constructor are evaluator and evaluator.pars that allow the specification of functions for calculating user-defined evaluation metrics. Finally, the last parameter of the constructor is trainReq that is a Boolean value indicating whether the training data should also be ``sent’’ to the evaluation functions. This is useful for metrics that require the training data for being calculated (e.g. some normalized metrics use the average value of the target variable in the training set).

powErr <- function(trues,preds,pow=3,...) {
    c(pow.err = mean((trues-preds)^pow))
}

EstimationTask(metrics="pow.err",method=CV(), 
               evaluator="powErr",evaluator.pars=list(pow=4))

EstimationTask(evaluator="powErr",evaluator.pars=list(pow=4))

## Task for estimating all metrics of the selected evaluation function using
##  1 x 10 - Fold Cross Validation
##   Run with seed =  1234

data(BreastCancer,package='mlbench')
library(e1071)
bcExp <- performanceEstimation(
  PredTask(Class ~ .,BreastCancer[,-1],'BreastCancer'),
  workflowVariants('standardWF',
           learner='svm',
           learner.pars=list(cost=c(1,5),gamma=c(0.01,0.1))
          ),
  EstimationTask(metrics=c("F","prec","rec"),
                 evaluator.pars=list(posClass="malignant"),
                 method=CV(nReps=3,nFolds=10,strat=TRUE)))

data(Servo,package='mlbench')
library(nnet)
nnExp <- performanceEstimation(
  PredTask(Class ~ .,Servo),
  workflowVariants(learner='nnet',
                   learner.pars=list(trace=F,linout=T,
                       size=c(3,5),decay=c(0.01,0.1))
          ),
  EstimationTask(metrics="mse",method=Bootstrap(nReps=100)))

data(LetterRecognition,package='mlbench')
ltrExp <- performanceEstimation(
    PredTask(lettr ~ .,LetterRecognition),
    workflowVariants(learner='rpartXse',
                     learner.pars=list(se=c(0,1)),
                     predictor.pars=list(type='class')
                     ),
    EstimationTask(metrics="err",method=Holdout(nReps=3,hldSz=0.3)))

data(iris)
library(e1071)
irisExp <- performanceEstimation(
    PredTask(Species ~ .,iris),
    workflowVariants(learner='svm',
                     learner.pars=list(cost=c(1,10))
                     ),
    EstimationTask(metrics="acc",method=LOOCV()))

library(quantmod)
library(randomForest)
GSPC <- getSymbols('^GSPC',from='2008-01-01',to='2012-12-31', auto.assign=FALSE)
data.model <- specifyModel(
  Next(100*Delt(Ad(GSPC))) ~ Delt(Ad(GSPC),k=1:10))
data <- modelData(data.model)
colnames(data)[1] <- 'PercVarClose'
spExp <- performanceEstimation(
  PredTask(PercVarClose ~ .,data,'SP500_2012'),
  c(Workflow('standardWF',wfID="standRF",
             learner='randomForest',learner.pars=list(ntree=500)),
    Workflow('timeseriesWF',wfID="slideRF",
             learner='randomForest',
             learner.pars=list(ntree=500,relearn.step=30))
   ),
  EstimationTask(metrics="theil",
                 method=MonteCarlo(nReps=10,szTrain=0.5,szTest=0.25)))

7 Statistical Significance of Differences

The estimation methodologies that we have presented in the previous sections allow the user to obtain estimates of the mean predictive performance of different workflows or variants of these workflows, on different predictive tasks. We have seen that by applying the `summary} method to the objects resulting from the estimation experiments we can obtain the average performance for each candidate workflow on each task. These numbers are estimates of the expected mean performance of the workflows on the respective tasks. Being estimates, the obvious next question is to check whether the observed differences in performance between the workflows are statistically significant. More formally, we want to know that confidence level of rejecting the null hypothesis that the difference between the estimated averages is zero.

That is the goal of the function pairedComparisons(). This function implements a series of statistical hypothesis tests that can be used in different types of setups. We follow the general recommendations of Demsar (Demsar 2006). The function pairedComparisons() implements a series of statistical tests that can be used under different conditions. The function calculates them all (whenever possible) and it is up to the user to use the ones that are more adequate to answer her/his research questions (see Demsar (Demsar 2006) for a series of guidelines on how to proceed).

Function pairedComparisons() returns a list with as many components as there are metrics estimated. For each metric another list includes the different tests that are carried out as well as several other information. Namely, for each error metric the returned list includes the following components:

• setup - with generic information on the estimation task
• avgScores - a matrix with the average scores on the metric of each workflow on each task
• medScores - another matrix with information like the previous one, but instead of the average we get the median scores
• rks - a matrix with the rank positions corresponding to the average scores
• avgRksWfs - a vector with the average of the above ranks across all tasks
• t.test - a list containing information concerning paired t tests
• WilcoxonSignedRank.test - a list containing information concerning paired Wilcoxon Signed Rank tests
• F.test - a list containing information concerning the F test
• Nemenyi.test - a list containing information concerning the post-hoc Nemenyi test
• BonferroniDunn.test - a list containing information concerning the post-hoc Bonferroni-Dunn test

For both the t and Wilcoxon Signed Rank tests the components contain and array with 3 dimensions, the third dimension being the task, which means that for each task you get a matrix of results of paired comparisons. Namely, paired comparisons between each workflow and the baseline workflow. Each row of the matrix contains the average (or median in the case of Wilcoxon) score of the respective workflow, the difference between this score and the score of the baseline workflow, and the p value associated with the hypothesis that this difference is distributed around mean (median in Wilcoxon). Please note that although our function calculates the results of the t test, this is not recommended for most settings given the lack of independence of the values on each iteration (for instance on k-fold cross validation each of the k scores is obtained with training sets that have a string overlap).

The other 3 tests can be used in the following two settings: (i) test if there are significant differences among any pair of workflows; and (ii) test the significance of the differences between all workflows against a baseline. In both of these two settings we should start by checking the results of the F test to check if we can reject the null hypothesis that the average ranks of all workflows are equivalent. In case this hypothesis is rejected (component rejNull of the list F.test) we can move to the post-hoc tests. In the setup (i) we should use the Nemenyi test to find the critical difference among ranks above which we can say that the respective difference is statistically significant. The Nemenyi.test component is a list with components containing this information (critical difference value, ranking differences among all pairs of workflows, and whether these differences are or not significant). Under the setup (ii) the post-hoc test that should be used is the Bonferroni-Dunn test. In this case we just want to check if the difference between the average rank of each workflow and that of the baseline is or not significant (thus much less paired comparisons). The result is again given as a list where we have information on the critical difference, the baseline workflow name, the vector of average differences and whether these differences are different or not.

Let us see a concrete example of using this statistical tests. Suppose we want to apply several variants of an SVM to three classification tasks and we want to check if there are significant differences among them. The following test carries out the performance estimation task:

library(e1071)
data(PimaIndiansDiabetes,package='mlbench')
data(iris)
data(Glass,package="mlbench")
res <- performanceEstimation(
    c(PredTask(diabetes ~ .,PimaIndiansDiabetes,"Pima"),
      PredTask(Type ~ ., Glass),
      PredTask(Species ~ .,iris)),
    workflowVariants(learner="svm",
                     learner.pars=list(cost=1:5,gamma=c(0.1,0.01,0.001))),
    EstimationTask(metrics="err",method=CV()))

The object res can be passed to pairedComparisons() to calculate the tests described before:

pres <- pairedComparisons(res)

As we have several workflows being compared on different tasks we will start by checking the results of the Friedman F test:

pres$err$F.test

## $chi
## [1] 18.575
## 
## $FF
## [1] 1.585912
## 
## $critVal
## [1] 0.8892237
## 
## $rejNull
## [1] TRUE

We can see that the null hypothesis that the average ranks of the workflows are equivalent can be rejected and thus we can proceed to the post-hoc test. In case we wish the check the hypothesis that there are statistically significant differences among the workflows we should proceed with a Nemenyi post-hoc test. The results of this test can be inspected as follows:

pres$err$Nemenyi.test

## $critDif
## [1] 12.38302
## 
## $rkDifs
##            svm.v1    svm.v2    svm.v3    svm.v4    svm.v5    svm.v6
## svm.v1  0.0000000 1.5000000 1.0000000 1.8333333 1.5000000 0.6666667
## svm.v2  1.5000000 0.0000000 0.5000000 0.3333333 0.0000000 2.1666667
## svm.v3  1.0000000 0.5000000 0.0000000 0.8333333 0.5000000 1.6666667
## svm.v4  1.8333333 0.3333333 0.8333333 0.0000000 0.3333333 2.5000000
## svm.v5  1.5000000 0.0000000 0.5000000 0.3333333 0.0000000 2.1666667
## svm.v6  0.6666667 2.1666667 1.6666667 2.5000000 2.1666667 0.0000000
## svm.v7  0.8333333 2.3333333 1.8333333 2.6666667 2.3333333 0.1666667
## svm.v8  0.1666667 1.3333333 0.8333333 1.6666667 1.3333333 0.8333333
## svm.v9  1.3333333 0.1666667 0.3333333 0.5000000 0.1666667 2.0000000
## svm.v10 2.8333333 1.3333333 1.8333333 1.0000000 1.3333333 3.5000000
## svm.v11 7.6666667 9.1666667 8.6666667 9.5000000 9.1666667 7.0000000
## svm.v12 6.6666667 8.1666667 7.6666667 8.5000000 8.1666667 6.0000000
## svm.v13 2.3333333 3.8333333 3.3333333 4.1666667 3.8333333 1.6666667
## svm.v14 1.0000000 2.5000000 2.0000000 2.8333333 2.5000000 0.3333333
## svm.v15 1.0000000 2.5000000 2.0000000 2.8333333 2.5000000 0.3333333
##            svm.v7    svm.v8    svm.v9   svm.v10   svm.v11  svm.v12
## svm.v1  0.8333333 0.1666667 1.3333333  2.833333  7.666667 6.666667
## svm.v2  2.3333333 1.3333333 0.1666667  1.333333  9.166667 8.166667
## svm.v3  1.8333333 0.8333333 0.3333333  1.833333  8.666667 7.666667
## svm.v4  2.6666667 1.6666667 0.5000000  1.000000  9.500000 8.500000
## svm.v5  2.3333333 1.3333333 0.1666667  1.333333  9.166667 8.166667
## svm.v6  0.1666667 0.8333333 2.0000000  3.500000  7.000000 6.000000
## svm.v7  0.0000000 1.0000000 2.1666667  3.666667  6.833333 5.833333
## svm.v8  1.0000000 0.0000000 1.1666667  2.666667  7.833333 6.833333
## svm.v9  2.1666667 1.1666667 0.0000000  1.500000  9.000000 8.000000
## svm.v10 3.6666667 2.6666667 1.5000000  0.000000 10.500000 9.500000
## svm.v11 6.8333333 7.8333333 9.0000000 10.500000  0.000000 1.000000
## svm.v12 5.8333333 6.8333333 8.0000000  9.500000  1.000000 0.000000
## svm.v13 1.5000000 2.5000000 3.6666667  5.166667  5.333333 4.333333
## svm.v14 0.1666667 1.1666667 2.3333333  3.833333  6.666667 5.666667
## svm.v15 0.1666667 1.1666667 2.3333333  3.833333  6.666667 5.666667
##          svm.v13   svm.v14   svm.v15
## svm.v1  2.333333 1.0000000 1.0000000
## svm.v2  3.833333 2.5000000 2.5000000
## svm.v3  3.333333 2.0000000 2.0000000
## svm.v4  4.166667 2.8333333 2.8333333
## svm.v5  3.833333 2.5000000 2.5000000
## svm.v6  1.666667 0.3333333 0.3333333
## svm.v7  1.500000 0.1666667 0.1666667
## svm.v8  2.500000 1.1666667 1.1666667
## svm.v9  3.666667 2.3333333 2.3333333
## svm.v10 5.166667 3.8333333 3.8333333
## svm.v11 5.333333 6.6666667 6.6666667
## svm.v12 4.333333 5.6666667 5.6666667
## svm.v13 0.000000 1.3333333 1.3333333
## svm.v14 1.333333 0.0000000 0.0000000
## svm.v15 1.333333 0.0000000 0.0000000
## 
## $signifDifs
##         svm.v1 svm.v2 svm.v3 svm.v4 svm.v5 svm.v6 svm.v7 svm.v8 svm.v9
## svm.v1   FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v2   FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v3   FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v4   FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v5   FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v6   FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v7   FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v8   FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v9   FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v10  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v11  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v12  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v13  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v14  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
## svm.v15  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE  FALSE
##         svm.v10 svm.v11 svm.v12 svm.v13 svm.v14 svm.v15
## svm.v1    FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v2    FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v3    FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v4    FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v5    FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v6    FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v7    FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v8    FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v9    FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v10   FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v11   FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v12   FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v13   FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v14   FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
## svm.v15   FALSE   FALSE   FALSE   FALSE   FALSE   FALSE

This is obviously a too extensive set of information though the conclusions are centered in the signifDifs component. An even better alternative can be obtained through CD diagrams, which we have implemented in function CDdiagram.Nemenyi whose result is shown in Figure~\ref{fig:cdn`.

CDdiagram.Nemenyi(pres)

The CD diagram of the Nemenyi post-hoc test.

At the top left of the diagram you can see a horizontal line with the critical difference for the differences between average ranks to be considered statistically significant. Then each workflow is represented by a line with origin in its respective average rank (in the X axis). If the lines of any pair of workflows are connected by a horizontal black line, it means that the difference between them is not statistically significant. In the diagram of Figure~\ref{fig:cdn` we can observe that all lines are connected meaning that none of the paired differences between all workflows can be seen as statistically significant.

8 Larger Examples

The main advantage of the infra-structure we are proposing is to automate large scale performance estimation experiments. It is on these very large setups that the use of the infra-structure saves more time to the user. However, in these contexts the objects resulting from the estimation process are very large and some of the tools we have shown before for exploring the results may produce over-cluttered output. In effect, if you have an experiment involving dozens of predictive tasks and eventually hundreds of workflow variants being compared on several evaluation metrics, doing a plot of the resulting object is simply not possible as the graph will be unreadable. This section illustrates some of these cases and presents some solutions to overcome the difficulties they bring.

Extremely large experiments may take days or weeks to complete, depending on the available hardware. In this context, it may not be wise to run the experiments on a single call to the performanceEstimation() function because if something goes wrong in the middle you may loose lots of work/time. Using the random number generation seeds that are available in all experimental settings objects we can split the experiments in several calls and still ensure that the same data folds are used in all estimation experiments. If the seeding process is not enough due to the usage of different hardware for instance, then you may still resort to the user supplied data splits to make sure all methods are compared on the same data. We will see that when all experiments are finished we will be able to merge the objects of each call into a single object as if we had issued a single call.

Another way of improving performance on large experiments is through parallel computation. includes facilities to run experiment in parallel through the existing parallel back-ends in R. The simplest of options, that has the advantage of not requiring you to know anything at all about parallel computation in R, is to call the performanceEstimation function with a fourth parameter that will make all experiments to run in parallel on the different cores of any standard multicore computer. The following is a simple example of achieving this effect:

library(performanceEstimation)
library(e1071)
data(Satellite,package="mlbench")
pres <- performanceEstimation(
    PredTask(classes ~ .,Satellite),
    Workflow(learner="svm"),
    EstimationTask("err",CV()),
    cluster=TRUE
    )

By adding this extra parameter the lower level functions that implement the estimation procedures (in this case cross validation but the concept is applicable to any of the other methods), will automatically create a (local) parallel back-end using half of the cores of your computer. On reasonably large data sets this simple extra setting will typically lead to cut execution times by a significant amount.

Other more complex parallel settings are also possible but they require you to know how to create the clusters before calling our function. For instance, you could create a cluster (using function makeCluster of package parallel) that involved several computers. The object of class cluster that results from these steps should then be sent through parameter cluster of our performanceEstimation function so that the function can take advantage of this parallel back-end. The following is a simple illustrative example:

library(parallel)
myclust <- makeCluster(c("localhost","192.168.2.10","192.168.2.13"),"SOCK")
library(performanceEstimation)
library(e1071)
data(Satellite,package="mlbench")
pres <- performanceEstimation(
    PredTask(classes ~ .,Satellite),
    Workflow(learner="svm"),
    EstimationTask("err",CV()),
    cluster=myclust
    )
stopCluster(myclust)

Please note that there are a few assumptions on the above example, namely concerning access of your username in localhost to the remote machines whose IP’s are given. This means that the username exists on all machines and that appropriate SSH keys have been set so that no passwords are required for accessing the remote machines.

Let us now focus on the issue of splitting a large experiment in partial calls to performanceEstimation() and at the end merge the separate results into a single object as if the experiments were run all together. Please note that further efficiency gains could be achieved if some parallel setup involving different computers as the one described above was used. The following example addresses several regression tasks using several workflows that are variants of different regression algorithms. We illustrate how to create the different PredTask objects programmatically, how to create the different workflow variants, and how to call performanceEstimation() with different task/workflow combinations storing the intermediate objects on temporary files for later reading and merging,

library(performanceEstimation)
library(e1071)
library(randomForest)

data(algae,package="DMwR")
DSs <- sapply(names(algae)[12:18],
         function(x,names.attrs) { 
           f <- as.formula(paste(x,"~ ."))
           PredTask(f,algae[,c(names.attrs,x)],x,copy=TRUE) 
         },
         names(algae)[1:11])

WFs <- list()
WFs$svm <- list(learner.pars=list(cost=c(10,150,300),
                                  gamma=c(0.01,0.001),
                                  epsilon=c(0.1,0.05)),
                pre="centralImp",post="na2central")
WFs$randomForest <- list(learner.pars=list(mtry=c(5,7),
                                           ntree=c(500,750,1500)),
                         pre="centralImp")

for(d in seq_along(DSs)) {
  for(w in names(WFs)) {
    resObj <- paste(names(DSs)[d],w,'Res',sep='')
    assign(resObj,
           performanceEstimation(
                  DSs[d],         
                  c(
                    do.call('workflowVariants',
                            c(list(learner=w),WFs[[w]]))
                    ),
                   EstimationTask(metrics=c('mse','mae'),method=CV(nReps=3)),
                   cluster=TRUE)
           )

    save(list=resObj,file=paste(names(DSs)[d],w,'Rdata',sep='.'))
  }
}

The above code compares 12 SVM variants with 6 random forest variants, on 7 algae blooms regression tasks, using \(3\times 10-\)fold cross validation. Although this is not a very large experimental comparison it still includes applying 18 different workflow variants on 7 different prediction tasks, 30 times, i.e. 3780 train+test cycles. Instead of running all these experiments on a single call to the function performanceEstimation() (which would obviously still be possible), we have made different calls for each workflow type (SVM and random forest) and for each predictive task. This means that each call will run all variants of a certain workflow on a certain predictive task. The result of each of these calls will be assigned to an object with a name composed of the task name and workflow learner (the resObj variable). In the end each of these objects is saved on a file with a similar name, for future loading and results analysis. For instance, in the end there will be a file with name a1.svm.Rdata'' which contains an object of class **ComparisonResults** named `a1svmRes`. This object contains the MSE and MAE estimated scores of the SVM variants on the task of predicting the target variablea1’’ (one of the seven algae in this data set).

Later on, after the above experiment has finished you can load the saved objects back into R and moreover, join them into a single object, as shown below:

nD <- paste('a',1:7,sep='')
nL <- c('svm','randomForest')
res <- NULL
for(d in nD) {
  resD <- NULL
  for(l in nL) {
    load(paste(d,l,'Rdata',sep='.'))
    x <- get(paste(d,l,'Res',sep=''))
    resD <- if (is.null(resD)) x else mergeEstimationRes(resD,x,by='workflows')
  }
  res <- if (is.null(res)) resD else mergeEstimationRes(res,resD,by='tasks')
}
save(res,file='allResultsAlgae.Rdata')

The mergeEstimationRes() function when applied to objects of class ComparisonResults allows merging of these objects across different dimensions. Namely, such objects have the individual scores of all experiments spread across 3 dimensions: the metrics, the workflows and the tasks. The argument by of the mergeEstimationRes() function allows you to specify how to merge the given objects. The most common situations are: (i) merging the results of different workflows over the same data sets - you should use by="workflows", or (ii) merging the results of the same workflows across different tasks - you should use by="tasks".

The following code can be used to check that the merging was OK, and also to illustrate a few other utility functions whose purpose should be obvious:

res

## 
## ==  Cross Validation Performance Estimation Experiment ==
## 
## Task for estimating  mse,mae  using
##  3 x 10 - Fold Cross Validation
##   Run with seed =  1234 
## 
##  18  workflows applied to  7  predictive tasks

taskNames(res)

## [1] "a1" "a2" "a3" "a4" "a5" "a6" "a7"

workflowNames(res)

##  [1] "svm.v1"          "svm.v2"          "svm.v3"         
##  [4] "svm.v4"          "svm.v5"          "svm.v6"         
##  [7] "svm.v7"          "svm.v8"          "svm.v9"         
## [10] "svm.v10"         "svm.v11"         "svm.v12"        
## [13] "randomForest.v1" "randomForest.v2" "randomForest.v3"
## [16] "randomForest.v4" "randomForest.v5" "randomForest.v6"

metricNames(res)

## [1] "mse" "mae"

With such large objects the most we can do is obtaining the best scores or rankings of the workflows:

topPerformers(res)

## $a1
##            Workflow Estimate
## mse randomForest.v3  263.869
## mae randomForest.v3   11.057
## 
## $a2
##            Workflow Estimate
## mse randomForest.v1   94.102
## mae         svm.v11    5.917
## 
## $a3
##     Workflow Estimate
## mse   svm.v3   44.442
## mae  svm.v10    3.911
## 
## $a4
##            Workflow Estimate
## mse randomForest.v3   18.343
## mae         svm.v10    1.882
## 
## $a5
##            Workflow Estimate
## mse randomForest.v1   44.609
## mae          svm.v7    4.037
## 
## $a6
##     Workflow Estimate
## mse   svm.v6  111.757
## mae  svm.v12    5.548
## 
## $a7
##            Workflow Estimate
## mse randomForest.v5   26.227
## mae         svm.v10    2.383

rankWorkflows(res)

## $a1
## $a1$mse
##          Workflow Estimate
## 1 randomForest.v3 263.8695
## 2 randomForest.v5 264.5268
## 3 randomForest.v1 264.7324
## 4 randomForest.v4 272.2932
## 5 randomForest.v6 272.7472
## 
## $a1$mae
##          Workflow Estimate
## 1 randomForest.v3 11.05689
## 2 randomForest.v1 11.06012
## 3 randomForest.v5 11.07450
## 4 randomForest.v2 11.22290
## 5 randomForest.v4 11.22579
## 
## 
## $a2
## $a2$mse
##          Workflow Estimate
## 1 randomForest.v1 94.10169
## 2 randomForest.v3 94.22759
## 3 randomForest.v5 94.28289
## 4 randomForest.v6 95.03010
## 5 randomForest.v4 95.06688
## 
## $a2$mae
##   Workflow Estimate
## 1  svm.v11 5.917035
## 2   svm.v1 5.937923
## 3  svm.v10 5.940801
## 4   svm.v4 5.954024
## 5   svm.v5 5.967910
## 
## 
## $a3
## $a3$mse
##          Workflow Estimate
## 1          svm.v3 44.44234
## 2          svm.v2 44.54107
## 3          svm.v8 45.27262
## 4          svm.v9 45.31149
## 5 randomForest.v3 46.47451
## 
## $a3$mae
##   Workflow Estimate
## 1  svm.v10 3.911241
## 2   svm.v4 3.914820
## 3   svm.v7 3.943484
## 4   svm.v1 3.956006
## 5   svm.v5 4.006819
## 
## 
## $a4
## $a4$mse
##          Workflow Estimate
## 1 randomForest.v3 18.34261
## 2 randomForest.v1 18.36200
## 3          svm.v4 18.36773
## 4 randomForest.v5 18.39560
## 5         svm.v10 18.58282
## 
## $a4$mae
##   Workflow Estimate
## 1  svm.v10 1.881994
## 2   svm.v4 1.883301
## 3   svm.v7 1.983666
## 4   svm.v1 1.994250
## 5  svm.v11 2.022148
## 
## 
## $a5
## $a5$mse
##          Workflow Estimate
## 1 randomForest.v1 44.60891
## 2 randomForest.v5 44.71305
## 3 randomForest.v3 44.75107
## 4 randomForest.v4 45.70916
## 5 randomForest.v2 45.74127
## 
## $a5$mae
##   Workflow Estimate
## 1   svm.v7 4.036624
## 2  svm.v10 4.055948
## 3   svm.v4 4.061444
## 4   svm.v1 4.074267
## 5   svm.v5 4.077211
## 
## 
## $a6
## $a6$mse
##          Workflow Estimate
## 1          svm.v6 111.7569
## 2         svm.v12 113.1213
## 3          svm.v5 121.9274
## 4         svm.v11 123.1705
## 5 randomForest.v5 123.4793
## 
## $a6$mae
##   Workflow Estimate
## 1  svm.v12 5.548443
## 2   svm.v7 5.572169
## 3   svm.v6 5.587264
## 4   svm.v4 5.633257
## 5   svm.v1 5.643838
## 
## 
## $a7
## $a7$mse
##          Workflow Estimate
## 1 randomForest.v5 26.22656
## 2 randomForest.v3 26.24998
## 3 randomForest.v1 26.29936
## 4 randomForest.v2 26.34936
## 5 randomForest.v4 26.44271
## 
## $a7$mae
##   Workflow Estimate
## 1  svm.v10 2.382957
## 2   svm.v4 2.407783
## 3   svm.v7 2.419313
## 4   svm.v1 2.439576
## 5   svm.v6 2.439598

Notice that both topPerformers() and rankWorkflows() assume that the evaluation metrics are to be minimized, i.e. they assume the lower the better the scores. Still, both functions have a parameter named maxs that accepts a vector with as many Boolean values as there are evaluation metrics being estimated, which you may use to indicate that some particular metric is to be maximized and not minimized (the default). So for instance, if you had an experiment where the 1st and 3rd metrics are to be minimized, whilst the second is to be maximized, you could call these functions as rankWorkflows(resObj,maxs=c(F,T,F)).

In order to obtain further results from these large objects one usually proceeds by analyzing parts of the object, for instance focusing on a particular task or metric, or even a subset of the workflows. To facilitate this we can use the generic function subset() that can also be applied to objects of class ComparisonResults. An example of its use is given below, which results in a graph of the performance of the different workflows in the predictive task a1'', in terms ofMAE’’, which is show in the next Figure:

plot(subset(res, tasks='a1', metrics='mae'))

The MAE results for the task A1.

As before we are using the generic function plot() but this time applied to a subset of the original object with all results. This subset is obtained using the generic function subset() that accepts several parameters to specify the subset we are interested on. In this case we are using the parameters tasks and metrics to indicate that we want to analyze only the results concerning the task a1'' and the metricmae’‘. Other possibility is the parameter workflows for indicating a subset of the workflows. Both workflows, tasks and metrics accept as values a character string containing a regular expression that will be used internally with the R function grep() over the vector of names of the respective objects (names of the workflows, names of the tasks and names of the metrics, respectively). For instance, if you want to constrain the previous graph even further to the workflows whose name ends in ``4’’ (absurd example of course!), you could use the following:

plot(subset(res, tasks='a1', workflows='4$'))

Illustration of the use of regular expressions in sub-setting the results objects.

If you are more familiar with the syntax of “wildcards” you may use the R function glob2rx() to convert to regular expressions, as show in the following example:

summary(subset(res, tasks='a1', workflows=glob2rx('*svm*'),metrics='mse'))

## 
## == Summary of a  Cross Validation Performance Estimation Experiment ==
## 
## Task for estimating  mse  using
##  3 x 10 - Fold Cross Validation
##   Run with seed =  1234 
## 
## * Predictive Tasks ::  a1
## * Workflows  ::  svm.v1, svm.v2, svm.v3, svm.v4, svm.v5, svm.v6, svm.v7, svm.v8, svm.v9, svm.v10, svm.v11, svm.v12 
## 
## -> Task:  a1
##   *Workflow: svm.v1 
##               mse
## avg     305.37037
## std     147.62784
## med     275.08382
## iqr     211.22466
## min      79.09407
## max     605.36798
## invalid   0.00000
## 
##   *Workflow: svm.v2 
##              mse
## avg     349.1888
## std     171.6171
## med     301.5451
## iqr     196.5386
## min     120.9873
## max     821.4265
## invalid   0.0000
## 
##   *Workflow: svm.v3 
##              mse
## avg     367.1953
## std     180.4110
## med     345.6525
## iqr     203.3054
## min     142.0827
## max     915.9872
## invalid   0.0000
## 
##   *Workflow: svm.v4 
##               mse
## avg     356.54033
## std     187.29360
## med     326.86720
## iqr     318.83908
## min      82.03737
## max     711.67485
## invalid   0.00000
## 
##   *Workflow: svm.v5 
##              mse
## avg     376.3833
## std     235.3738
## med     282.9879
## iqr     308.7286
## min     104.9834
## max     975.8439
## invalid   0.0000
## 
##   *Workflow: svm.v6 
##                mse
## avg      396.15700
## std      287.60697
## med      285.41018
## iqr      294.37250
## min       98.40015
## max     1260.52110
## invalid    0.00000
## 
##   *Workflow: svm.v7 
##               mse
## avg     304.23863
## std     144.75696
## med     271.52274
## iqr     212.00615
## min      78.69374
## max     587.86503
## invalid   0.00000
## 
##   *Workflow: svm.v8 
##              mse
## avg     355.5680
## std     176.4419
## med     313.3674
## iqr     190.9779
## min     130.2359
## max     862.8762
## invalid   0.0000
## 
##   *Workflow: svm.v9 
##              mse
## avg     375.2226
## std     188.0744
## med     350.1804
## iqr     195.8608
## min     143.0326
## max     954.8950
## invalid   0.0000
## 
##   *Workflow: svm.v10 
##               mse
## avg     359.45788
## std     188.57821
## med     331.28706
## iqr     320.80755
## min      82.84897
## max     721.55851
## invalid   0.00000
## 
##   *Workflow: svm.v11 
##              mse
## avg     367.1249
## std     216.2313
## med     282.4140
## iqr     288.6100
## min     101.2520
## max     959.5912
## invalid   0.0000
## 
##   *Workflow: svm.v12 
##                mse
## avg      376.05576
## std      244.87929
## med      284.02493
## iqr      278.97164
## min       94.36646
## max     1179.99337
## invalid    0.00000

The following are some illustrations of the use of other available utility functions.

Obtaining the scores on all iterations and metrics of a workflow on a particular task:

getScores(res, 'svm.v6','a3')

##           mse      mae
## 1   58.773058 4.039651
## 2   63.420588 4.836262
## 3  124.212723 6.629852
## 4   20.818437 2.902345
## 5   28.507875 3.202152
## 6   18.424942 2.774288
## 7   14.446194 2.509190
## 8   54.975336 4.584341
## 9   23.521575 2.871156
## 10 135.025071 6.735133
## 11 112.234104 5.328625
## 12  81.117165 5.569711
## 13  34.504072 3.779135
## 14   9.790665 2.162704
## 15  30.126166 3.476657
## 16  58.715226 4.654773
## 17  23.793327 3.171367
## 18  26.480963 3.236722
## 19  99.572929 5.928280
## 20  27.569010 3.293526
## 21   8.495628 2.129290
## 22  27.933235 3.197012
## 23  20.164491 2.850632
## 24  72.373337 5.364697
## 25  69.470293 4.813790
## 26 126.075926 5.477111
## 27  65.167023 5.459279
## 28   9.437532 2.299811
## 29  22.696736 3.511973
## 30  90.988401 5.114133

Getting the summary of the results of a particular workflow on a predictive task :

estimationSummary(res,'svm.v3', 'a7')

##                mse      mae
## avg      30.516360 3.156927
## std      29.265924 1.213391
## med      23.096086 3.016725
## iqr      31.305352 1.053218
## min       3.083724 1.393694
## max     141.576530 7.615879
## invalid   0.000000 0.000000

Finally, the metricsSummary() function allows you to apply any summary function (defaulting to mean()) to the iterations estimates. The following calculates the median of the results of the SVMs on the task ``a1’’,

metricsSummary(subset(res, workflows=glob2rx('*svm*'), tasks='a1'),
               summary='median')

## $a1
##        svm.v1    svm.v2    svm.v3    svm.v4    svm.v5    svm.v6    svm.v7
## mse 275.08382 301.54507 345.65254 326.86720 282.98787 285.41018 271.52274
## mae  11.93685  12.65768  13.78127  12.85777  12.21762  12.10388  11.82759
##        svm.v8    svm.v9   svm.v10   svm.v11   svm.v12
## mse 313.36744 350.18044 331.28706 282.41397 284.02493
## mae  12.89971  13.67427  12.87902  12.13263  12.10956

9 Conclusions

We have presented performanceEstimation that aims at being a general package for estimating and comparing the performance of any approach to any predictive task in R. The package allows users to very easily carry out standard out-of-the-box comparative experiments between existing modeling tools in R. However, it also allows the users to supply their own functions implementing special workflows to solve tasks. This means that the package should cover the needs of occasional users as well as advanced users that which to try and compared their own proposed workflows.

The performanceEstimation also includes several facilities for testing the statistical significance of the observed differences, namely implementing the current state of the art in this subject as described in (Demsar 2006), including CD diagrams for both the Nemenyi and Bonferroni-Dunn post-hoc tests.

Finally, we have provided a few illustrations of how to use performanceEstimation for larger experiments, namely taking advantage of parallel computation that is available in R.