% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/nanoparquet-package.R
\name{parquet-encodings}
\alias{parquet-encodings}
\title{Parquet encodings}
\description{
Various Parquet encodings
}
\section{Nanoparquet defaults}{
Currently the defaults are decided based on the R types. This might
change in the future. In general, the defaults will likely change until
nanoparquet reaches version 1.0.0.

Current encoding defaults:
\itemize{
\item Definition levels always use \code{RLE}. (Nanoparquet does not currently
write repetition levels, but they'll also use \code{RLE}, once implemented.)
\item \code{factor} columns use \code{RLE_DICTIONARY}.
\item \code{logical} columns use \code{RLE} if the average run length of the first
10,000 values is at least 15. Otherwise they use the \code{PLAIN} encoding.
\item \code{integer}, \code{double} and \code{character} columns use \code{RLE_DICTIONARY} if at
least two third of their values are repeated. Otherwise they use
\code{PLAIN} encoding.
\item \code{list} columns of \code{raw} vectors always use the \code{PLAIN} encoding
currently.
}
}

\section{Parquet encodings}{
See \url{https://github.com/apache/parquet-format/blob/master/Encodings.md}
for more details on Parquet encodings.
\subsection{\code{PLAIN} encoding}{

Supported types: all.

In general values are written back to back:
\itemize{
\item Integer types are little endian.
\item Floating point types follow the IEEE standard.
\item \code{BYTE_ARRAY}: for each element, there is a little endian 4-byte length
and then the bytes themselves.
\item \code{FIXED_LEN_BYTE_ARRAY}: bytes are written back to back.
}

Nanoparquet can read and write this encoding for all primitive types.
}

\subsection{\code{RLE_DICTIONARY} encoding}{

Supported types: dictionary indices in data pages.

This encoding combines run-length encoding and bit-packing.
Repeated sequences of the same value can be run-length encoded, and
non-repeated parts are bit packed.
It is used for data pages of dictionaries.
The dictionary pages themselves are \code{PLAIN} encoded.

The deprecated \code{PLAIN_DICTIONARY} name is treated the same as
\code{RLE_DICTIONARY}.

Nanoparquet can read and write this encoding.
}

\subsection{\code{RLE} encoding}{

Supported types: \code{BOOLEAN}. Also for definition and repetition levels.

This is the same encoding as \code{RLE_DICTIONARY}, with a slightly different
header. It combines run-length encoding and bit packing.
It is used for \code{BOOLEAN} columns, and also for definition and
repetition levels.

Nanoparquet can read and write this encoding.
}

\subsection{\code{BIT_PACKED} encoding (deprecated in favor of \code{RLE})}{

Supported types: none. Only for definition and repetition levels, but
\code{RLE} should be used instead.

This is a simple bit packing encoding for integers, that was previously
used for encoding definition and repetition levels. It is not used in new
Parquet files because the the \code{RLE} encoding includes it and it is better.

Nanoparquet currently cannot read or write the \code{BIT_PACKED} encoding.
}

\subsection{\code{DELTA_BINARY_PACKED} encoding}{

Supported types: \code{INT32}, \code{INT64}.

This encoding efficiently encodes integer columns if the differences
between consecutive elements are often the same, and/or the differences
between consecutive elements are small. The extreme case of an arithmetic
sequence can be encoded in O(1) space.

Nanoparquet can read this encoding, but cannot currently write it.
}

\subsection{\code{DELTA_LENGTH_BYTE_ARRAY} encoding}{

Supported types: \code{BYTE_ARRAY}.

This encoding uses \code{DELTA_BINARY_PACKED} to encode the length of all
byte array elements. It is especially efficient for short byte array
elements, i.e. a column of short strings.

Nanoparquet can read this encoding, but cannot currently write it.
}

\subsection{\code{DELTA_BYTE_ARRAY} encoding}{

Supported types: \code{BYTE_ARRAY}, \code{FIXED_LEN_BYTE_ARRAY}.

This encoding is efficient if consecutive byte array elements share the
same prefix, because each element can reuse a prefix of the previous
element.

Nanoparquet can read this encoding, but cannot currently write it.
}

\subsection{\code{BYTE_STREAM_SPLIT} encoding}{

Supported types: \code{FLOAT}, \code{DOUBLE}, \code{INT32}, \code{INT64},
\code{FIXED_LEN_BYTE_ARRAY}.

This encoding stores the first bytes of the elements first, then the
second bytes, etc. It does not reduce the size in itself, but may allow
more efficient compression.

Nanoparquet can read this encoding, but cannot currently write it.
}
}

\seealso{
\code{\link[=write_parquet]{write_parquet()}} on how to select a non-default encoding when
writing Parquet files.
}
