# zarrs Backend

pizzarr ships in two tiers. The CRAN build is pure R — no Rust compilation, no system dependencies. It handles local and HTTP Zarr stores with sequential chunk I/O via `lapply`. The r-universe build compiles in the zarrs Rust crate via extendr, adding parallel decompression, cloud-native store backends (S3, GCS), and codecs beyond what R packages provide.

The split exists because CRAN’s macOS build machines ship a Rust toolchain (rustc 1.84) that is too old for zarrs, which requires rustc >= 1.91. r-universe builds against the latest stable toolchain, so it can compile zarrs and distribute pre-built binaries. End users on either tier install with `install.packages()` — no Rust toolchain needed.

## Checking availability

    library(pizzarr)

    has_zarrs <- pizzarr:::.pizzarr_env$zarrs_available

`pizzarr_compiled_features()` lists the feature flags compiled into the zarrs backend. On the CRAN tier it returns `character(0)` with a message; on the r-universe tier it returns the compiled capabilities:

    pizzarr_compiled_features()
    #> zarrs backend not available (pure R install).
    #> See ?pizzarr_upgrade for the r-universe install.
    #> character(0)

The internal flag `.pizzarr_env$zarrs_available` is a logical scalar set once at package load. Dispatch logic throughout pizzarr checks this flag to decide whether to call into Rust or fall through to the R-native path:

    pizzarr:::.pizzarr_env$zarrs_available
    #> [1] FALSE

## Upgrading to the zarrs tier

`pizzarr_upgrade()` prints the r-universe install command when zarrs is not compiled in, or confirms that the backend is already present:

    pizzarr_upgrade()
    #> Install pizzarr with the zarrs backend from r-universe:
    #>
    #>   install.packages("pizzarr", repos = "https://zarr-developers.r-universe.dev")

The startup message that CRAN users see on `library(pizzarr)` can be silenced with `options(pizzarr.suggest_runiverse = FALSE)`.

## Probing store metadata

The examples below require the zarrs backend. When this vignette is built without it, the code chunks are not evaluated.

`zarrs_node_exists()` opens a filesystem store via the Rust backend, probes for V2 and V3 metadata keys at a given path, and returns a list with three fields: `exists` (logical), `node_type` (character), and `zarr_format` (integer or NULL). The store handle is cached on the Rust side — subsequent calls to the same store path reuse it without re-opening.

### V2 store

    v2_root <- pizzarr_sample("fixtures/v2/data.zarr")

    # Root group
    zarrs_node_exists(v2_root, "")

    # An array within the store
    zarrs_node_exists(v2_root, "1d.contiguous.lz4.i2")

    # A path that does not exist
    zarrs_node_exists(v2_root, "does_not_exist")

### V3 store

V2 and V3 detection is automatic. zarrs probes for `zarr.json` first (V3), then falls back to `.zarray` / `.zgroup` (V2):

    v3_root <- pizzarr_sample("fixtures/v3/data.zarr")

    zarrs_node_exists(v3_root, "")

## Store cache management

The Rust backend holds open store handles in a process-global cache keyed by normalized path. `zarrs_close_store()` removes a handle from the cache and returns `TRUE`. A second call to the same path returns `FALSE` — it was already removed:

    zarrs_close_store(v2_root)
    zarrs_close_store(v2_root)

    zarrs_close_store(v3_root)

## Array metadata

`zarrs_open_array_metadata()` opens a zarrs array and returns its metadata as a named list. The store handle is cached, so repeated calls to the same store are fast. The returned list contains `shape`, `chunks`, `dtype`, `r_type`, `fill_value_json`, `zarr_format`, and `order`.

### V2 array

    v2_root <- pizzarr_sample("fixtures/v2/data.zarr")
    zarrs_open_array_metadata(v2_root, "1d.contiguous.raw.i2")

### V3 array

V3 arrays work the same way. The `zarr_format` field distinguishes V2 from V3:

    v3_root <- pizzarr_sample("fixtures/v3/data.zarr")
    zarrs_open_array_metadata(v3_root, "1d.contiguous.gzip.i2")

### Data type classification

The `r_type` field maps zarrs data types to R-compatible type families. zarrs numeric types are classified as `"double"`, `"integer"`, or `"logical"` based on what R can represent natively:

- double: float64 (zero-cost), float32 (widened), uint32/int64/uint64 (widened, precision risk > 2^53)
- integer: int32 (zero-cost), int8/int16/uint8/uint16 (widened)
- logical: bool

Unsupported types (strings, complex) report `"unsupported"` and fall back to the R-native code path.

    zarrs_close_store(v2_root)
    zarrs_close_store(v3_root)

## Runtime info and tuning

`zarrs_runtime_info()` reports the current zarrs configuration — the codec concurrency target, thread pool size, how many store handles are cached, and which features were compiled in:

    zarrs_runtime_info()

### pizzarr_config()

`pizzarr_config()` is the main interface for viewing and changing concurrency settings. Called with no arguments it returns the current state; with arguments it sets the specified values:

    # View current settings
    pizzarr_config()

    # Set codec concurrency to 2 parallel operations per read/write
    pizzarr_config(concurrent_target = 2L)
    zarrs_runtime_info()$codec_concurrent_target

Three settings are available:

- nthreads — rayon thread pool size. Set-once per R session (the thread pool can only be initialised once). For reliable session-level control, set the `PIZZARR_NTHREADS` environment variable before starting R.
- concurrent_target — how many codec operations zarrs runs in parallel within a single read or write call. Can be changed at any time.
- http_batch_range_requests — whether HTTP stores use multipart range requests (default TRUE). Set to FALSE for servers with incomplete multipart support. Takes effect on the next store open.

All three settings can also be configured via environment variables (`PIZZARR_NTHREADS`, `PIZZARR_CONCURRENT_TARGET`, `PIZZARR_HTTP_BATCH_RANGE_REQUESTS`) or R options (`pizzarr.nthreads`, etc.), which are read at package load time. Environment variables persist across sessions without needing `.Rprofile` edits.

The lower-level `zarrs_set_codec_concurrent_target()` function is still available for direct use:

    zarrs_set_codec_concurrent_target(2L)
    zarrs_runtime_info()$codec_concurrent_target

## Reading data via zarrs

When the zarrs backend is available and the selection is a contiguous slice (step == 1), `ZarrArray$get_item()` dispatches reads to zarrs automatically. zarrs handles chunk identification, parallel decompression, and codec execution internally, bypassing pizzarr’s R-native chunk loop. Scalar integer selections (e.g., selecting a single row of a matrix) are also eligible — they become length-1 ranges on the Rust side. Unsupported selections (step > 1 slices, fancy indexing, MemoryStore) fall through to the R-native path transparently.

### Basic read

    d <- tempfile("zarrs_vignette_")
    z <- zarr_create(store = d, shape = c(100L, 50L), chunks = c(10L, 10L),
                     dtype = "<f8")
    z$set_item("...", array(as.double(seq_len(5000)), dim = c(100, 50)))

    # Re-open and read a subset --- zarrs handles the chunk I/O
    z2 <- zarr_open(store = d)
    result <- z2$get_item(list(slice(1L, 10L), slice(1L, 5L)))
    dim(result$data)

### Direct zarrs_get_subset call

For lower-level access, `zarrs_get_subset()` reads a contiguous subset directly via the Rust backend. Ranges are 0-based with exclusive stop, matching zarrs conventions:

    result <- zarrs_get_subset(d, "", list(c(0L, 10L), c(0L, 5L)), NULL)
    str(result)

### Concurrency control

The optional `concurrent_target` parameter (or the `pizzarr.concurrent_target` R option) controls how many parallel codec operations zarrs uses within a single read call. Setting it to `1L` disables parallel decompression:

    result <- zarrs_get_subset(d, "", list(c(0L, 10L), c(0L, 5L)), 1L)
    length(result$data)

    zarrs_close_store(d)
    unlink(d, recursive = TRUE)

## Creating arrays via zarrs

When the zarrs backend is available and the store is a writable filesystem path, `zarr_create()` dispatches array creation to zarrs instead of building metadata JSON in R. zarrs validates the metadata structure, writes it to the store, and the array is ready for data. The dispatch is transparent — the same `zarr_create()` call works on both tiers, and unsupported configurations (MemoryStore, object dtypes, custom filters) fall through to the R-native path.

### Transparent dispatch

The `zarr_create()` examples earlier in this vignette already use this path when zarrs is available. The zarrs backend handles V2 and V3 formats, all 11 numeric data types, and four codec presets:

    # V3 array with gzip compression
    d <- tempfile("zarrs_create_vignette_")
    z <- zarr_create(store = d, shape = c(20L, 10L), chunks = c(10L, 10L),
                     dtype = "<f8", zarr_format = 3L)
    z

    # Confirm V3 metadata was written
    file.exists(file.path(d, "zarr.json"))

    zarrs_close_store(d)
    unlink(d, recursive = TRUE)

### Direct zarrs_create_array call

`zarrs_create_array()` provides lower-level access to the Rust creation path. It accepts V3-style data type names (`"float64"`, `"int32"`, `"bool"`, etc.) and a codec preset string (`"none"`, `"gzip"`, `"blosc"`, or `"zstd"`). The return value is the same metadata list as `zarrs_open_array_metadata()`:

    d <- tempfile("zarrs_create_direct_")
    dir.create(d)

    meta <- zarrs_create_array(
      store_url = d,
      array_path = "",
      shape = c(100L, 50L),
      chunks = c(10L, 10L),
      dtype = "float64",
      codec_preset = "gzip",
      fill_value = 0.0,
      attributes_json = "{}",
      zarr_format = 3L
    )
    str(meta)

The array is immediately usable for reads and writes:

    zarrs_set_subset(d, "", list(c(0L, 10L), c(0L, 5L)),
                     as.double(1:50), NULL)
    result <- zarrs_get_subset(d, "", list(c(0L, 10L), c(0L, 5L)), NULL)
    head(result$data)

    zarrs_close_store(d)
    unlink(d, recursive = TRUE)

### Codec presets

The zarrs creation path supports four named codec presets. Custom codec configurations fall through to the R-native path.

Preset | V2 compressor | V3 codec chain | Notes
--- | --- | --- | ---
`"none"` | null | bytes only | No compression
`"gzip"` | gzip, level 1 | bytes + gzip(1) | Fast, reasonable ratio
`"blosc"` | blosc, lz4, clevel 5 | bytes + blosc(lz4, 5) | Requires `blosc` feature
`"zstd"` | — | bytes + zstd(3) | V3 only; requires `zstd` feature

One difference from the R-native path: zarrs uses the `"gzip"` compressor id for V2 arrays, while zarr-python uses `"zlib"`. Both produce gzip-compatible output, and zarrs reads either id when opening existing arrays.

## Writing data via zarrs

The write path mirrors the read path. When the zarrs backend is available and the selection qualifies (contiguous slices, filesystem-backed store), `ZarrArray$set_item()` dispatches writes to zarrs instead of iterating over chunks in R. zarrs encodes the data, splits it across the affected chunks, and writes them to disk — using its internal thread pool for parallel compression when multiple chunks are involved.

Data type narrowing happens on the Rust side. R doubles narrow to the array’s stored type (float32, int64, uint32, etc.) and R integers narrow to smaller integer types (int16, int8, uint8, uint16) with range checking. An out-of-range value produces an error rather than silent truncation.

### Basic write

    d <- tempfile("zarrs_write_vignette_")
    z <- zarr_create(store = d, shape = c(20L, 10L), chunks = c(10L, 10L),
                     dtype = "<f8")

    # set_item dispatches to zarrs when eligible
    z$set_item("...", array(as.double(1:200), dim = c(20, 10)))

    # Read back to confirm
    z2 <- zarr_open(store = d)
    result <- z2$get_item(list(slice(1L, 5L), slice(1L, 3L)))
    result$data

### Partial overwrite

Writing to a subset of an existing array works the same way. zarrs reads the affected chunks, merges the new data, and writes them back:

    # Overwrite rows 3-7, columns 1-2
    z$set_item(list(slice(3L, 7L), slice(1L, 2L)),
               array(rep(-1.0, 10), dim = c(5, 2)))

    result <- z2$get_item(list(slice(1L, 10L), slice(1L, 3L)))
    result$data

### Direct zarrs_set_subset call

`zarrs_set_subset()` provides lower-level access to the Rust write path. Data is a flat vector in R’s native F-order (column-major) — the Rust backend handles the F-to-C order conversion internally. The function returns `TRUE` on success:

    # Write 10 values to the first row (0-based range [0, 1) x [0, 10))
    zarrs_set_subset(d, "", list(c(0L, 1L), c(0L, 10L)),
                       as.double(101:110), NULL)

    result <- zarrs_get_subset(d, "", list(c(0L, 1L), c(0L, 10L)), NULL)
    result$data

    zarrs_close_store(d)
    unlink(d, recursive = TRUE)

## HTTP reads via zarrs

When the `http_sync` feature is compiled in, the zarrs backend can read directly from HTTP/HTTPS Zarr stores using the zarrs_http crate. This bypasses pizzarr’s R-native `crul`-based chunk loop, giving parallel chunk decode on remote data.

HTTP stores are read-only in zarrs — write dispatch (`set_item`) falls through to the R-native path automatically.

### Transparent dispatch

The zarrs fast path activates automatically when an `HttpStore`-backed array is read with a contiguous selection. No code changes are needed compared to the R-native path:

    url <- "https://raw.githubusercontent.com/DOI-USGS/rnz/main/inst/extdata/bcsd.zarr"

    z <- zarr_open(store = HttpStore$new(url))

    # zarrs handles the HTTP reads + parallel decompression
    pr <- z$get_item("pr")
    pr

    # Read a subset --- zarrs fetches only the chunks that overlap
    result <- pr$get_item(list(slice(1L, 3L), slice(1L, 5L), slice(1L, 5L)))
    dim(result$data)

### Direct zarrs_get_subset from HTTP

`zarrs_get_subset()` also works with HTTP URLs. The store handle is cached on the Rust side, so repeated reads to the same URL reuse the connection:

    meta <- zarrs_open_array_metadata(url, "pr")
    str(meta[c("shape", "dtype", "zarr_format")])

    # Read a single element (first along each dimension)
    ranges <- lapply(seq_along(meta$shape), function(i) c(0L, 1L))
    result <- zarrs_get_subset(url, "pr", ranges, NULL)
    result$data

    zarrs_close_store(url)

### Feature detection

Check whether HTTP support is compiled in with `pizzarr_compiled_features()`. When `"http_sync"` is present, zarrs can open `http://` and `https://` URLs. When it is absent, HTTP reads fall through to the R-native `crul`-based path:

    "http_sync" %in% pizzarr_compiled_features()

## S3 reads via zarrs

When the `s3` feature is compiled in, the zarrs backend can read from Amazon S3 buckets using the object_store crate with an async-to-sync adapter. Public buckets work without credentials (unsigned requests). Authenticated access uses standard AWS environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`).

S3 stores are currently read-only via zarrs — write operations fall through to the R-native path.

    # OME-Zarr bonsai dataset on AWS Open Data (V2, zstd, uint8)
    s3_url <- "s3://ome-zarr-scivis/v0.4/64x0/bonsai.ome.zarr"

    # Read array metadata
    meta <- zarrs_open_array_metadata(s3_url, "scale0/bonsai")
    str(meta[c("shape", "dtype", "zarr_format")])

    # Read a small subset (first 4x4x4 corner)
    result <- zarrs_get_subset(s3_url, "scale0/bonsai",
                               list(c(0L, 4L), c(0L, 4L), c(0L, 4L)), NULL)
    str(result)

    zarrs_close_store(s3_url)

### GCS and other cloud stores

GCS data hosted on Google Cloud Storage is publicly accessible via HTTPS endpoints. The zarrs HTTP backend reads these directly:

    # Pangeo ECCO ocean basins (V2, blosc/lz4, float32)
    gcs_url <- "https://storage.googleapis.com/pangeo-data/ECCO_basins.zarr"

    meta <- zarrs_open_array_metadata(gcs_url, "basin_mask")
    cat("Shape:", paste(meta$shape, collapse = " x "), "\n")
    cat("Dtype:", meta$dtype, "\n")

    # Read a single basin mask slice
    result <- zarrs_get_subset(gcs_url, "basin_mask",
                               list(c(0L, 1L), c(0L, 90L), c(0L, 90L)), NULL)
    cat("Slice dimensions:", paste(result$shape, collapse = " x "), "\n")

    zarrs_close_store(gcs_url)

Authenticated GCS access via `gs://` URLs requires the `gcs` compiled feature and GCP credentials (environment variables or application default credentials). The `S3Store` and `GcsStore` R6 classes provide URL wrappers for high-level use with `zarr_open()`:

    # S3 (requires s3 feature)
    z <- zarr_open(store = S3Store$new("s3://bucket/path/to/store.zarr"))

    # GCS (requires gcs feature + credentials)
    z <- zarr_open(store = GcsStore$new("gs://bucket/path/to/store.zarr"))

## C/F order handling

zarrs stores data in C-order (row-major), while R uses F-order (column-major). The Rust backend handles this conversion transparently:

- Reads: `zarrs_get_subset()` returns data in F-order, ready for `array(data, dim = shape)` with no `aperm()` needed.
- Writes: `zarrs_set_subset()` accepts F-order data and converts to C-order internally before writing to the store.

The transpose uses cache-blocked tiling for 2D arrays and output-order iteration with incremental index tracking for higher dimensions, matching or exceeding the performance of R’s C-level `aperm()`.