% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/convert.R
\name{spod_convert}
\alias{spod_convert}
\title{Convert data from plain text to duckdb or parquet format}
\usage{
spod_convert(
  type = c("od", "origin-destination", "os", "overnight_stays", "nt", "number_of_trips"),
  zones = c("districts", "dist", "distr", "distritos", "municipalities", "muni",
    "municip", "municipios"),
  dates = NULL,
  save_format = "duckdb",
  save_path = NULL,
  overwrite = FALSE,
  data_dir = spod_get_data_dir(),
  quiet = FALSE,
  max_mem_gb = NULL,
  max_n_cpu = max(1, parallelly::availableCores() - 1),
  max_download_size_gb = 1,
  ignore_missing_dates = FALSE
)
}
\arguments{
\item{type}{The type of data to download. Can be \code{"origin-destination"} (or ust \code{"od"}), or \code{"number_of_trips"} (or just \code{"nt"}) for v1 data. For v2 data \code{"overnight_stays"} (or just \code{"os"}) is also available. More data types to be supported in the future. See codebooks for v1 and v2 data in vignettes with \code{spod_codebook(1)} and \code{spod_codebook(2)} (\link{spod_codebook}).}

\item{zones}{The zones for which to download the data. Can be \code{"districts"} (or \code{"dist"}, \code{"distr"}, or the original Spanish \code{"distritos"}) or \code{"municipalities"} (or \code{"muni"}, \code{"municip"}, or the original Spanish \code{"municipios"}) for both data versions. Additionaly, these can be \code{"large_urban_areas"} (or \code{"lua"}, or the original Spanish \code{"grandes_areas_urbanas"}, or \code{"gau"}) for v2 data (2022 onwards).}

\item{dates}{A \code{character} or \code{Date} vector of dates to process. Kindly keep in mind that v1 and v2 data follow different data collection methodologies and may not be directly comparable. Therefore, do not try to request data from both versions for the same date range. If you need to compare data from both versions, please refer to the respective codebooks and methodology documents. The v1 data covers the period from 2020-02-14 to 2021-05-09, and the v2 data covers the period from 2022-01-01 to the present until further notice. The true dates range is checked against the available data for each version on every function run.

The possible values can be any of the following:
\itemize{
\item For the \code{spod_get()} and \code{spod_convert()} functions, the \code{dates} can be set to "cached_v1" or "cached_v2" to request data from cached (already previously downloaded) v1 (2020-2021) or v2 (2022 onwards) data. In this case, the function will identify and use all data files that have been downloaded and cached locally, (e.g. using an explicit run of \code{spod_download()}, or any data requests made using the \code{spod_get()} or \code{spod_convert()} functions).
\item A single date in ISO (YYYY-MM-DD) or YYYYMMDD format. \code{character} or \code{Date} object.
\item A vector of dates in ISO (YYYY-MM-DD) or YYYYMMDD format. \code{character} or \code{Date} object. Can be any non-consecutive sequence of dates.
\item A date range
\itemize{
\item eigher a \code{character} or \code{Date} object of length 2 with clearly named elements \code{start} and \code{end} in ISO (YYYY-MM-DD) or YYYYMMDD format. E.g. \code{c(start = "2020-02-15", end = "2020-02-17")};
\item or a \code{character} object of the form \code{YYYY-MM-DD_YYYY-MM-DD} or \code{YYYYMMDD_YYYYMMDD}. For example, \verb{2020-02-15_2020-02-17} or \verb{20200215_20200217}.
}
\item A regular expression to match dates in the format \code{YYYYMMDD}. \code{character} object. For example, \verb{^202002} will match all dates in February 2020.
}}

\item{save_format}{A \code{character} vector of length 1 with values "duckdb" or "parquet". Defaults to "duckdb". If \code{NULL} automatically inferred from the \code{save_path} argument. If only \code{save_format} is provided, \code{save_path} will be set to the default location set in \code{SPANISH_OD_DATA_DIR} environment variable using \link{spod_set_data_dir}\code{(path = 'path/to/your/cache/dir')}. So for v1 data that path would be \verb{<data_dir>/clean_data/v1/tabular/duckdb/} or \verb{<data_dir>/clean_data/v1/tabular/parquet/}.

You can also set \code{save_path}. If it ends with ".duckdb", will save to \code{DuckDB} database format, if \code{save_path} does not end with ".duckdb", will save to \code{parquet} format and will treat the \code{save_path} as a path to a folder, not a file, will create necessary hive-style subdirectories in that folder. Hive style looks like \code{year=2020/month=2/day=14} and inside each such directory there will be a \code{data_0.parquet} file that contains the data for that day.}

\item{save_path}{A \code{character} vector of length 1. The full (not relative) path to a \code{DuckDB} database file or \code{parquet} folder.
\itemize{
\item If \code{save_path} ends with \code{.duckdb}, it will be saved as a DuckDB database file. The format argument will be automatically set to \code{save_format='duckdb'}.
\item If \code{save_path} ends with a folder name (e.g. \verb{/data_dir/clean_data/v1/tabular/parquet/od_distr} for origin-destination data for district level), the data will be saved as a collection of \code{parquet} files in a hive-style directory structure. So the subfolders of \code{od_distr} will be \code{year=2020/month=2/day=14} and inside each of these folders a single \code{parquet} file will be placed containing the data for that day.
\item If \code{NULL}, uses the default location in \code{data_dir} (set by the \code{SPANISH_OD_DATA_DIR} environment variable using \link{spod_set_data_dir}\code{(path = 'path/to/your/cache/dir')}. Therefore, the default relative path for \code{DuckDB} is \verb{<data_dir>/clean_data/v1/tabular/duckdb/<type>_<zones>.duckdb} and for \code{parquet} files is \verb{<data_dir>/clean_data/v1/tabular/parquet/<type>_<zones>/}, where \code{type} is the type of data (e.g. 'od', 'os', 'nt', that correspoind to 'origin-destination', 'overnight-stays', 'number-of-trips', etc.) and \code{zones} is the name of the geographic zones (e.g. 'distr', 'muni', etc.). See the details below in the function arguments description.
}}

\item{overwrite}{A \code{logical} or a \code{character} vector of length 1. If \code{TRUE}, overwrites existing \code{DuckDB} or \code{parquet} files. Defaults to \code{FALSE}. For parquet files can also be set to 'update', so that only parquet files are only created for the dates that have not yet been converted.}

\item{data_dir}{The directory where the data is stored. Defaults to the value returned by \code{spod_get_data_dir()} which returns the value of the environment variable \code{SPANISH_OD_DATA_DIR} or a temporary directory if the variable is not set. To set the data directory, use \link{spod_set_data_dir}.}

\item{quiet}{A \code{logical} value indicating whether to suppress messages. Default is \code{FALSE}.}

\item{max_mem_gb}{\code{integer} value of the maximum operating memory to use in GB. \code{NULL} by default, delegates the choice to the \code{DuckDB} engine which usually sets it to 80\% of available memory. Caution, in HPC use, the amount of memory available to your job may be determined incorrectly by the \code{DuckDB} engine, so it is recommended to set this parameter explicitly according to your job's memory limits.}

\item{max_n_cpu}{The maximum number of threads to use. Defaults to the number of available cores minus 1.}

\item{max_download_size_gb}{The maximum download size in gigabytes. Defaults to 1.}

\item{ignore_missing_dates}{Logical. If \code{TRUE}, the function will not raise an error if the some of the specified dates are missing. Any dates that are missing will be skipped, however the data for any valid dates will be acquired. Defaults to \code{FALSE}.}
}
\value{
Path to saved \code{DuckDB} database file or to a folder with \code{parquet} files in hive-style directory structure.
}
\description{
\ifelse{html}{\href{https://lifecycle.r-lib.org/articles/stages.html#stable}{\figure{lifecycle-stable.svg}{options: alt='[Stable]'}}}{\strong{[Stable]}}

Converts data for faster analysis into either \code{DuckDB} file or into \code{parquet} files in a hive-style directory structure. Running analysis on these files is sometimes 100x times faster than working with raw CSV files, espetially when these are in gzip archives. To connect to converted data, please use 'mydata <- \link{spod_connect}(data_path = path_returned_by_spod_convert)' passing the path to where the data was saved. The connected \code{mydata} can be analysed using \code{dplyr} functions such as \link[dplyr]{select}, \link[dplyr]{filter}, \link[dplyr]{mutate}, \link[dplyr]{group_by}, \link[dplyr]{summarise}, etc. In the end of any sequence of commands you will need to add \link[dplyr]{collect} to execute the whole chain of data manipulations and load the results into memory in an R \code{data.frame}/\code{tibble}. For more in-depth usage of such data, please refer to DuckDB documentation and examples at \url{https://duckdb.org/docs/api/r#dbplyr} . Some more useful examples can be found here \url{https://arrow-user2022.netlify.app/data-wrangling#combining-arrow-with-duckdb} . You may also use \code{arrow} package to work with parquet files \url{https://arrow.apache.org/docs/r/}.

For detailed data descriptions, see package vignettes using \code{\link[=spod_codebook]{spod_codebook(ver = 1)}} and \code{\link[=spod_codebook]{spod_codebook(ver = 2)}} and official methodology documents in \strong{References} section.
}
\examples{
\dontshow{if (interactive()) withAutoprint(\{ # examplesIf}
\donttest{
# Set data dir for file downloads
spod_set_data_dir(tempdir())

# download and convert data
dates_1 <- c(start = "2020-02-17", end = "2020-02-18")
db_2 <- spod_convert(
  type = "number_of_trips",
  zones = "distr",
  dates = dates_1,
  overwrite = TRUE
)

# now connect to the converted data
my_od_data_2 <- spod_connect(db_2)

# disconnect from the database
spod_disconnect(my_od_data_2)
}
\dontshow{\}) # examplesIf}
}
\references{
\itemize{
\item \strong{For the official website of the mobility study}: \insertRef{mitms_mobility_web}{spanishoddata}
\item \strong{For v1 data methodology}: \insertRef{mitma_methodology_2020_v3}{spanishoddata}
\item \strong{For v2 data methodology}: \insertRef{mitms_methodology_2022_v8}{spanishoddata}
\item \strong{For the \code{spanishoddata} R package}: \insertRef{spanishoddata-r-pkg}{spanishoddata}
}

Use \code{\link[=spod_cite]{spod_cite()}} to cite the package and the data with correct plain text, markdown, or BibTeX formats.
}
