% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/Sql.R
\name{renderTranslateQueryApplyBatched}
\alias{renderTranslateQueryApplyBatched}
\title{Render, translate, and perform process to batches of data.}
\usage{
renderTranslateQueryApplyBatched(
  connection,
  sql,
  fun,
  args = list(),
  errorReportFile = file.path(getwd(), "errorReportSql.txt"),
  snakeCaseToCamelCase = FALSE,
  tempEmulationSchema = getOption("sqlRenderTempEmulationSchema"),
  integerAsNumeric = getOption("databaseConnectorIntegerAsNumeric", default = TRUE),
  integer64AsNumeric = getOption("databaseConnectorInteger64AsNumeric", default = TRUE),
  ...
)
}
\arguments{
\item{connection}{The connection to the database server created using either
\code{\link[=connect]{connect()}} or \code{\link[DBI:dbConnect]{DBI::dbConnect()}}.}

\item{sql}{The SQL to be send.}

\item{fun}{Function to apply to batch. Must take data.frame and integer position as parameters.}

\item{args}{List of arguments to be passed to function call.}

\item{errorReportFile}{The file where an error report will be written if an error occurs.
Defaults to 'errorReportSql.txt' in the current working directory.}

\item{snakeCaseToCamelCase}{If true, field names are assumed to use snake_case, and are converted to camelCase.}

\item{tempEmulationSchema}{Some database platforms like Oracle and Impala do not truly support temp tables. To
emulate temp tables, provide a schema with write privileges where temp tables
can be created.}

\item{integerAsNumeric}{Logical: should 32-bit integers be converted to numeric (double) values? If FALSE
32-bit integers will be represented using R's native \code{Integer} class.}

\item{integer64AsNumeric}{Logical: should 64-bit integers be converted to numeric (double) values? If FALSE
64-bit integers will be represented using \code{bit64::integer64}.}

\item{...}{Parameters that will be used to render the SQL.}
}
\value{
Invisibly returns a list of outputs from each call to the provided function.
}
\description{
This function renders, and translates SQL, sends it to the server, processes the data in batches with a call back
function. Note that this function should perform a row-wise operation. This is designed to work with massive data
that won't fit in to memory.

The batch sizes are determined by the java virtual machine and will depend on the data.
}
\details{
Fields will be automatically converted for improved consistenty in these situations:
\itemize{
\item SQLite: Fields with names ending in \verb{_date} will be converted to DATE fields. Rationale: SQLite
does not support DATE fields.
\item SQLite: Fields with names ending in \verb{_datetime} will be converted to POSIXct fields. Rationale:
SQLite does not support DATETIME fields.
\item BigQuery and Snowflake: Integer fields will be converted to Integer if it fits in an integer, or
will remain Integer64 otherwise. Rationale: these platforms do not distinguish between INT and
BIGINT.
}

This function calls the \code{render} and \code{translate} functions in the \code{SqlRender} package before
calling \code{\link[=querySql]{querySql()}}.
}
\examples{
\dontrun{
connectionDetails <- createConnectionDetails(
  dbms = "postgresql",
  server = "localhost",
  user = "root",
  password = "blah",
  schema = "cdm_v4"
)
connection <- connect(connectionDetails)

# First example: write data to a large CSV file:
filepath <- "myBigFile.csv"
writeBatchesToCsv <- function(data, position, ...) {
  write.csv(data, filepath, append = position != 1)
  return(NULL)
}
renderTranslateQueryApplyBatched(connection,
  "SELECT * FROM @schema.person;",
  schema = "cdm_synpuf",
  fun = writeBatchesToCsv
)

# Second example: write data to Andromeda
# (Alternative to querySqlToAndromeda if some local computation needs to be applied)
bigResults <- Andromeda::andromeda()
writeBatchesToAndromeda <- function(data, position, ...) {
  data$p <- EmpiricalCalibration::computeTraditionalP(data$logRr, data$logSeRr)
  if (position == 1) {
    bigResults$rrs <- data
  } else {
    Andromeda::appendToTable(bigResults$rrs, data)
  }
  return(NULL)
}
sql <- "SELECT target_id, comparator_id, log_rr, log_se_rr FROM @schema.my_results;"
renderTranslateQueryApplyBatched(connection,
  sql,
  fun = writeBatchesToAndromeda,
  schema = "my_results",
  snakeCaseToCamelCase = TRUE
)

disconnect(connection)
}

}
