% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/tune_race_anova.R
\name{tune_race_anova}
\alias{tune_race_anova}
\alias{tune_race_anova.model_spec}
\alias{tune_race_anova.workflow}
\title{Efficient grid search via racing with ANOVA models}
\usage{
tune_race_anova(object, ...)

\method{tune_race_anova}{model_spec}(
  object,
  preprocessor,
  resamples,
  ...,
  param_info = NULL,
  grid = 10,
  metrics = NULL,
  eval_time = NULL,
  control = control_race()
)

\method{tune_race_anova}{workflow}(
  object,
  resamples,
  ...,
  param_info = NULL,
  grid = 10,
  metrics = NULL,
  eval_time = NULL,
  control = control_race()
)
}
\arguments{
\item{object}{A \code{parsnip} model specification or a \code{\link[workflows:workflow]{workflows::workflow()}}.}

\item{...}{Not currently used.}

\item{preprocessor}{A traditional model formula or a recipe created using
\code{\link[recipes:recipe]{recipes::recipe()}}. This is only required when \code{object} is not a workflow.}

\item{resamples}{An \code{rset()} object that has multiple resamples (i.e., is not
a validation set).}

\item{param_info}{A \code{\link[dials:parameters]{dials::parameters()}} object or \code{NULL}. If none is given,
a parameters set is derived from other arguments. Passing this argument can
be useful when parameter ranges need to be customized.}

\item{grid}{A data frame of tuning combinations or a positive integer. The
data frame should have columns for each parameter being tuned and rows for
tuning parameter candidates. An integer denotes the number of candidate
parameter sets to be created automatically.}

\item{metrics}{A \code{\link[yardstick:metric_set]{yardstick::metric_set()}} or \code{NULL}. The first metric in
metrics is the one that will be optimized.}

\item{eval_time}{A numeric vector of time points where dynamic event time
metrics should be computed (e.g. the time-dependent ROC curve, etc). The
values must be non-negative and should probably be no greater than the
largest event time in the training set (See Details below).}

\item{control}{An object used to modify the tuning process. See
\code{\link[=control_race]{control_race()}} for more details.}
}
\value{
An object with primary class \code{tune_race} in the same standard format
as objects produced by \code{\link[tune:tune_grid]{tune::tune_grid()}}.
}
\description{
\code{\link[=tune_race_anova]{tune_race_anova()}} computes a set of performance metrics (e.g. accuracy or RMSE)
for a pre-defined set of tuning parameters that correspond to a model or
recipe across one or more resamples of the data. After an initial number of
resamples have been evaluated, the process eliminates tuning parameter
combinations that are unlikely to be the best results using a repeated
measure ANOVA model.
}
\details{
The technical details of this method are described in Kuhn (2014).

Racing methods are efficient approaches to grid search. Initially, the
function evaluates all tuning parameters on a small initial set of
resamples. The \code{burn_in} argument of \code{\link[=control_race]{control_race()}} sets the number of
initial resamples.

The performance statistics from these resamples are analyzed to determine
which tuning parameters are \emph{not} statistically different from the current
best setting. If a parameter is statistically different, it is excluded from
further resampling.

The next resample is used with the remaining parameter combinations and the
statistical analysis is updated. More candidate parameters may be excluded
with each new resample that is processed.

This function determines statistical significance using a repeated measures ANOVA
model where the performance statistic (e.g., RMSE, accuracy, etc.) is the
outcome data and the random effect is due to resamples. The
\code{\link[=control_race]{control_race()}} function contains are parameter for the significance cutoff
applied to the ANOVA results as well as other relevant arguments.

There is benefit to using racing methods in conjunction with parallel
processing. The following section shows a benchmark of results for one
dataset and model.
\subsection{Censored regression models}{

With dynamic performance metrics (e.g. Brier or ROC curves), performance is
calculated for every value of \code{eval_time} but the \emph{first} evaluation time
given by the user (e.g., \code{eval_time[1]}) is analyzed during racing.

Also, values of \code{eval_time} should be less than the largest observed event
time in the training data. For many non-parametric models, the results beyond
the largest time corresponding to an event are constant (or \code{NA}).
}

\subsection{Benchmarking results}{

To demonstrate, we use a SVM model with the \code{kernlab} package.

\if{html}{\out{<div class="sourceCode r">}}\preformatted{library(kernlab)
library(tidymodels)
library(finetune)
library(doParallel)

## -----------------------------------------------------------------------------

data(cells, package = "modeldata")
cells <- cells |> select(-case)

## -----------------------------------------------------------------------------

set.seed(6376)
rs <- bootstraps(cells, times = 25)
}\if{html}{\out{</div>}}

We’ll only tune the model parameters (i.e., not recipe tuning):

\if{html}{\out{<div class="sourceCode r">}}\preformatted{## -----------------------------------------------------------------------------

svm_spec <-
  svm_rbf(cost = tune(), rbf_sigma = tune()) |>
  set_engine("kernlab") |>
  set_mode("classification")

svm_rec <-
  recipe(class ~ ., data = cells) |>
  step_YeoJohnson(all_predictors()) |>
  step_normalize(all_predictors())

svm_wflow <-
  workflow() |>
  add_model(svm_spec) |>
  add_recipe(svm_rec)

set.seed(1)
svm_grid <-
  svm_spec |>
  parameters() |>
  grid_latin_hypercube(size = 25)
}\if{html}{\out{</div>}}

We’ll get the times for grid search and ANOVA racing with and without
parallel processing:

\if{html}{\out{<div class="sourceCode r">}}\preformatted{## -----------------------------------------------------------------------------
## Regular grid search

system.time(\{
  set.seed(2)
  svm_wflow |> tune_grid(resamples = rs, grid = svm_grid)
\})
}\if{html}{\out{</div>}}

\if{html}{\out{<div class="sourceCode">}}\preformatted{##    user  system elapsed 
## 741.660  19.654 761.357 
}\if{html}{\out{</div>}}

\if{html}{\out{<div class="sourceCode r">}}\preformatted{## -----------------------------------------------------------------------------
## With racing

system.time(\{
  set.seed(2)
  svm_wflow |> tune_race_anova(resamples = rs, grid = svm_grid)
\})
}\if{html}{\out{</div>}}

\if{html}{\out{<div class="sourceCode">}}\preformatted{##    user  system elapsed 
## 133.143   3.675 136.822 
}\if{html}{\out{</div>}}

Speed-up of 5.56-fold for racing.

\if{html}{\out{<div class="sourceCode r">}}\preformatted{## -----------------------------------------------------------------------------
## Parallel processing setup

cores <- parallel::detectCores(logical = FALSE)
cores
}\if{html}{\out{</div>}}

\if{html}{\out{<div class="sourceCode">}}\preformatted{## [1] 10
}\if{html}{\out{</div>}}

\if{html}{\out{<div class="sourceCode r">}}\preformatted{cl <- makePSOCKcluster(cores)
registerDoParallel(cl)
}\if{html}{\out{</div>}}

\if{html}{\out{<div class="sourceCode r">}}\preformatted{## -----------------------------------------------------------------------------
## Parallel grid search

system.time(\{
  set.seed(2)
  svm_wflow |> tune_grid(resamples = rs, grid = svm_grid)
\})
}\if{html}{\out{</div>}}

\if{html}{\out{<div class="sourceCode">}}\preformatted{##  user  system elapsed 
## 1.112   0.190 126.650 
}\if{html}{\out{</div>}}

Parallel processing with grid search was 6.01-fold faster than
sequential grid search.

\if{html}{\out{<div class="sourceCode r">}}\preformatted{## -----------------------------------------------------------------------------
## Parallel racing

system.time(\{
  set.seed(2)
  svm_wflow |> tune_race_anova(resamples = rs, grid = svm_grid)
\})
}\if{html}{\out{</div>}}

\if{html}{\out{<div class="sourceCode">}}\preformatted{##  user  system elapsed 
## 1.908   0.261  21.442 
}\if{html}{\out{</div>}}

Parallel processing with racing was 35.51-fold faster than sequential
grid search.

There is a compounding effect of racing and parallel processing but its
magnitude depends on the type of model, number of resamples, number of
tuning parameters, and so on.
}
}
\examples{
\donttest{
library(parsnip)
library(rsample)
library(dials)

## -----------------------------------------------------------------------------

if (rlang::is_installed(c("discrim", "lme4", "modeldata"))) {
  library(discrim)
  data(two_class_dat, package = "modeldata")

  set.seed(6376)
  rs <- bootstraps(two_class_dat, times = 10)

  ## -----------------------------------------------------------------------------

  # optimize an regularized discriminant analysis model
  rda_spec <-
    discrim_regularized(frac_common_cov = tune(), frac_identity = tune()) |>
    set_engine("klaR")

  ## -----------------------------------------------------------------------------

  ctrl <- control_race(verbose_elim = TRUE)
  set.seed(11)
  grid_anova <-
    rda_spec |>
    tune_race_anova(Class ~ ., resamples = rs, grid = 10, control = ctrl)

  # Shows only the fully resampled parameters
  show_best(grid_anova, metric = "roc_auc", n = 2)

  plot_race(grid_anova)
}
}
}
\references{
Kuhn, M 2014. "Futility Analysis in the Cross-Validation of Machine Learning
Models." \url{https://arxiv.org/abs/1405.6974}.
}
\seealso{
\code{\link[tune:tune_grid]{tune::tune_grid()}}, \code{\link[=control_race]{control_race()}}, \code{\link[=tune_race_win_loss]{tune_race_win_loss()}}
}
