% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/fd_fetch.R
\name{fd_fetch}
\alias{fd_fetch}
\title{Fetch source rows that are not yet present in the features table}
\usage{
fd_fetch(con, sql, key, feat_table_name, use_max_key = FALSE, verbose = FALSE)
}
\arguments{
\item{con}{A live \code{DBIConnection}, created with \code{\link[DBI:dbConnect]{DBI::dbConnect()}}.}

\item{sql}{A single SQL string (typically a \code{SELECT}) that defines the \strong{source
dataset} you want to process into features. In other words, it should return
the rows you would normally feed into your feature creation pipeline.
The query may join multiple tables and apply filters.

The query must be usable as a derived table: \verb{FROM (<sql>) AS r}.
Trailing semicolons are tolerated and removed.}

\item{key}{Name of the primary key column (character scalar). Must exist in
both the result of \code{sql} and in \code{feat_table_name}. This is the identifier used to
decide whether a source row has already been processed into the features table.}

\item{feat_table_name}{Name of the existing features table in the database
(character scalar). Must exist. This table is used only to identify which
\code{key} values are already present.}

\item{use_max_key}{Logical. If \code{TRUE}, \code{fd_fetch()} may reduce join work by
first computing \code{MAX(feat_table_name.key)} and selecting rows with \code{key > max_key}
(guaranteed not in \code{feat_table_name}). If \code{FALSE}, \code{fd_fetch()} selects all the rows
with any key no in \code{feat_table_name.key} which may be computationally more expensive.}

\item{verbose}{Logical. If \code{TRUE}, prints the executed SQL.}
}
\value{
A \code{data.frame} containing only rows returned by \code{sql} whose \code{key} is not
present in \code{feat_table_name}. The result includes an attribute \code{attr(x, "fd_fetch")}
(a list) with metadata such as \code{key}, \code{feat_table_name}, \code{use_max_key}, \code{max_key}
(if computed), \code{executed_sql}, and \code{n_rows}.
}
\description{
\code{fd_fetch()} executes a user-supplied SQL \code{SELECT} query against a database and
returns \strong{only} those rows whose primary key (\code{key}) is \strong{not present} in an
existing features table (\code{feat_table_name}).
}
\details{
This function is intentionally not a general-purpose query runner. It always
applies a "not yet processed" filter against \code{feat_table_name} and errors if that
cannot be done honestly (e.g., missing tables/columns).
\subsection{Important assumption}{

\code{fd_fetch()} assumes that:
\itemize{
\item the \code{sql} query is executed on the \strong{same database} (same \code{con}) where
\code{feat_table_name} is stored, and
\item the returned \code{key} values are comparable to \code{feat_table_name.key}.
}

Cross-database fetching (e.g., pulling data from one database and comparing to
a features table stored in another database) is not supported, because the
"not present in \code{feat_table_name}" filter must be evaluated by the database engine
in a single query context.
}
}
\examples{
if (requireNamespace("RSQLite", quietly = TRUE)) {
con <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")
on.exit(DBI::dbDisconnect(con), add = TRUE)

DBI::dbExecute(con, "CREATE TABLE raw (id INTEGER, x INTEGER)")
DBI::dbExecute(con, "CREATE TABLE r_variables_table (id INTEGER)")
DBI::dbExecute(con, "INSERT INTO raw (id, x) VALUES (1,10), (2,20), (3,30), (4,40), (5,50)")
DBI::dbExecute(con, "INSERT INTO r_variables_table (id) VALUES (1), (2), (4)")

# Returns ids 3 and 5 only (rows not yet present in the features table)
new_rows <- fd_fetch(
  con = con,
  sql = "SELECT * FROM raw",
  key = "id",
  feat_table_name = "r_variables_table"
)
}

}
