% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/createDisMatrix.R
\name{createDisMatrix}
\alias{createDisMatrix}
\title{Create a Dissimilarity Matrix from an Ensemble Model}
\usage{
createDisMatrix(
  ensemble,
  data,
  label,
  parallel = list(active = FALSE, no_cores = 1),
  verbose = FALSE,
  chunk_size = NULL,
  memory_limit = NULL,
  use_disk = FALSE,
  temp_dir = tempdir(),
  batch_aggregate = 10
)
}
\arguments{
\item{ensemble}{is an ensemble tree object}

\item{data}{is a data frame containing the variables in the model. It is the
data frame used for ensemble learning.}

\item{label}{is a character. It indicates the response label.}

\item{parallel}{A list with two elements: \code{active} (logical) and
\code{no_cores} (integer). If \code{active = TRUE}, the function performs
parallel computation using the number of cores specified in
\code{no_cores}. If \code{no_cores} is NULL or equal to 0, it defaults to
using all available cores minus one. If \code{active = FALSE}, the function
runs on a single core. Default: \code{list(active = FALSE, no_cores = 1)}.}

\item{verbose}{Logical. If TRUE, the function prints progress messages and
other information during execution. If FALSE (the default), messages are
suppressed.}

\item{chunk_size}{Integer. Number of rows to process in each chunk. If NULL,
automatically determined based on available memory and dataset size.
Default: NULL (auto).}

\item{memory_limit}{Numeric. Maximum memory to use in GB. Default: NULL (no limit).}

\item{use_disk}{Logical. If TRUE and dataset is very large, intermediate results
are saved to disk. Default: FALSE.}

\item{temp_dir}{Character. Directory for temporary files if use_disk = TRUE.
Default: tempdir().}

\item{batch_aggregate}{Integer. Number of tree results to aggregate at once
before adding to main matrix (reduces memory peaks). Default: 10.}
}
\value{
A dissimilarity matrix. This is a dissimilarity matrix measuring the
discordance between two observations concerning a given random forest
model.
}
\description{
The function createDisMatrix creates a dissimilarity matrix among
observations from an ensemble tree. This optimized version is designed
for large datasets (50K-500K observations) with improved memory management
and chunking capabilities.
}
\details{
This optimized version implements several strategies for handling
large datasets:

\itemize{
\item \strong{Memory-efficient aggregation}: Results from parallel trees are
aggregated in batches to avoid memory peaks
\item \strong{Chunking}: For very large matrices, computation can be split into
manageable chunks
\item \strong{Sparse matrix optimization}: Maintains sparsity throughout computation
\item \strong{Automatic garbage collection}: Explicit memory cleanup at critical points
\item \strong{Disk-based computation}: Optional saving of intermediate results for
datasets exceeding memory capacity
}

Supported ensemble types for \emph{classification} or \emph{regression} tasks:
\itemize{
\item \code{randomForest}
\item \code{ranger}
\item \code{xgb.Booster} (xgboost)
\item \code{lgb.Booster} (lightgbm)
\item \code{gbm} (gbm)
\item \code{catboost.CatBoost} (catboost)
}
}
\section{Interpretation note (RF vs boosting)}{

For \emph{bagging} ensembles (\code{randomForest}, \code{ranger}) the trees are
grown independently on bootstrap samples; co-occurrence in the same leaf
captures local similarity in the predictor space. For \emph{boosting} ensembles
(\code{xgb.Booster}, \code{lgb.Booster}, \code{gbm}, \code{catboost})
each tree is fit to the residual of the previous ones, so leaf
co-occurrence reflects similarity in the \emph{error-correction trajectory}
rather than in the final prediction space. The resulting dissimilarity
matrices therefore have systematically different scales (typically
\eqn{\bar D \in [0.85, 0.95]} for bagging vs. \eqn{[0.35, 0.70]} for
boosting). The surrogate tree built on top of \code{D} should be
interpreted accordingly.

The returned matrix carries an \code{ensemble_backend} attribute identifying
the backend used, which downstream functions check to detect mismatched
\code{(D, ensemble)} pairs.
}

\examples{
\donttest{
data("iris")

# Create training and validation set:
smp_size <- floor(0.75 * nrow(iris))
train_ind <- sample(seq_len(nrow(iris)), size = smp_size)
training <- iris[train_ind, ]
validation <- iris[-train_ind, ]
response_training <- training[,5]
response_validation <- validation[,5]

# Perform training:
## "randomForest" package
ensemble <- randomForest::randomForest(Species ~ ., data=training,
importance=TRUE, proximity=TRUE)

## "ranger" package
if (requireNamespace("ranger", quietly = TRUE)) {
  ensemble <- ranger::ranger(Species ~ ., data = iris,
    num.trees = 1000, importance = 'impurity')
}

# Compute dissimilarity matrix with optimizations
D <- createDisMatrix(
  ensemble,
  data = training,
  label = "Species",
  parallel = list(active = FALSE, no_cores = 1),
  chunk_size = 10000,  # Process 10K rows at a time
  batch_aggregate = 20, # Aggregate 20 trees at once
  verbose = TRUE
)
}

}
