API and Language Bindings · Paimon Vector Index
Description: Unified Paimon Vector Index APIs for Rust, C, C++, Java, and Python, including warm-up and metadata filter pushdown.

Skip to content

One lifecycle · five language surfaces

# API and language bindings

The Rust core, C FFI, C++ RAII layer, Java/JNI, and Python ctypes package share one train → write → serialize → detect → search model. Build options select the index type; Readers discover it from the file magic.

Unified lifecyclePositional I/OSingle and batch searchRoaring64 prefilter

## Public integration layers

Module | Role | Primary entry point
--- | --- | ---
`core` | Pure Rust indexes, file Readers/Writers, and benchmarks | `paimon_vindex_core::index`
`ffi` | C ABI over the Rust core; builds `libpaimon_vindex_ffi` | Generated `include/paimon_vindex.h`
`include` | C++ RAII wrapper | `paimon_vindex.hpp`
`jni` + `java` | JNI implementation and Java API | `org.apache.paimon.index.vector`
`python` | Pure Python package loading the C FFI through ctypes | `paimon_vindex`

The top-level Cargo workspace contains `core`, `ffi`, and `jni`. The Python package loads the shared FFI library at runtime.

## Shared lifecycle

01Create a Trainer
Parse and validate options
02Submit one or more
training batches
03Finish training and
create a one-shot Writer
04Add row IDs / vectors
and write the file
05Detect file magic
and execute searches

- Vectors are contiguous `f32` values; length must equal `vector_count × dimension`.
- Training data may arrive in batches. Every IVF trainer keeps a deterministic reservoir of at most `max(65,536, 64 × resolved nlist)` vectors. DiskANN starts from a 50,000-row cap and lowers it when necessary so the retained sample, optional cosine-normalized copy, codebook, and parallel PQ-training scratch fit `diskann.memory-budget-bytes`. Sampling is independent of batch boundaries.
- The Python and Java one-shot `train` helpers infer `dimension` from the matrix and use its row count for automatic `nlist`. When the matrix is only a sample, pass the final corpus size as `expected-vector-count`. Streaming Trainer APIs require a concrete dimension before their first batch.
- A Writer may receive production vectors in multiple batches. Row-ID count must equal vector count.
- Readers expose metadata, single-query search, batch search, and Roaring64-filtered variants.
- Files carry their type and resolved model sections. Callers do not pass index options again when opening a Reader.

## Shared search parameters

Parameter | Applies to | Description
--- | --- | ---
`top_k` | All indexes | Number of nearest neighbors returned for each query.
`Auto` width | All indexes | IVF starts at `max(8, ceil(nlist/16))`, adds enough average-list capacity for at least `4 × top_k` candidates, scales for filter selectivity, and progressively doubles when a filtered result is short. DiskANN uses a calibrated width when present, otherwise `max(100, 2 × top_k)`.
`nprobe` | IVF families | Explicit expert override for the number of lists probed. A tagged IVF width is rejected by DiskANN rather than silently ignored.
`l_search` | DiskANN | Explicit expert override for graph search-list size; it is clamped to at least `top_k`. A tagged DiskANN width is rejected by IVF indexes.

Use the binding's `automatic` search-parameter constructor. For DiskANN, `calibrate_search_width` / `calibrateSearchWidth` evaluates widths 100, 200, and 400 on representative queries and remembers the smallest adjacent pair with at least 98% Top-K result overlap. This is a stability proxy, not a ground-truth recall guarantee; explicit widths always win.

IVF-RQ width is a build option`rq.bits` accepts 1–8 and defaults to 4. It is persisted in the file and reported as `rq_bits` / `rqBits` metadata; search has no separate bit-width switch.

## Reader options for DiskANN

Reader options are accepted by every binding and affect DiskANN only. Other index families continue to use their existing positional-I/O behavior.

Concept | Rust / Python | C / C++ / Java | Default
--- | --- | --- | ---
Random-read latency hint | Input capability `estimated_random_read_latency_nanos` | C/C++ field / Java `estimatedRandomReadLatencyNanos()` | 0: reuse the mandatory header read's elapsed time; positive values bypass measurement
Total Reader memory | `memory_budget_bytes` | `memory_budget_bytes` / constructor argument | 4 GiB; automatically partitioned among required resident state, a profile-sized adjacency prefix, a bounded cold-adjacency LRU, and a bounded raw-vector LRU

The Reader has no public storage-tier switch. DiskANN selects an internal tier once during `open` from the latency hint or mandatory header-read timing; its latency, window, and beam policy then remain stable. `read_plan` / `readPlan` exposes that policy together with the current effective preload and shared-cache capacities. Those capacities are zero before resident initialization and may shrink when lazy row-ID lookup state consumes the same total budget. Cache partitioning remains an internal decision. The input callback receives all positional ranges for one round together and should execute them concurrently.

A storage adapter may additionally advertise `preferred_window_bytes` and a maximum range count named `max_ranges_per_pread` in Rust or `max_ranges_per_read` in C/C++/Python. Zero means unspecified. DiskANN rounds the requested window to complete 4 KiB logical pages, bounds it to 1 MiB, limits each `pread` batch to the advertised range count, and keeps physical alignment concerns inside the storage adapter. Build-time `deployment-profile` remains separate because it can select the persisted compact or interleaved layout. See DiskANN reader tuning.

## Search warm-up

After opening a Reader and before repeated searches, initialize resident state and, for DiskANN, optionally replay a small representative query set. Warm-up builds process-local caches without changing the file or search results.

Language | Resident initialization | Representative-query warm-up | DiskANN width calibration
--- | --- | --- | ---
Rust | `optimize_for_search` | `warmup_queries` | `calibrate_search_width`
C | `paimon_vindex_reader_optimize_for_search` | `paimon_vindex_reader_warmup_queries` | `paimon_vindex_reader_calibrate_search_width`
C++ | `optimize_for_search` | `warmup_queries` | `calibrate_search_width`
Java | `optimizeForSearch` | `warmupQueries` | `calibrateSearchWidth`
Python | `optimize_for_search` | `warmup_queries` | `calibrate_search_width`

`optimize_for_search` is optional for correctness: the first search performs the same initialization lazily. It builds IVF-PQ residual-L2 tables or DiskANN resident PQ/row-ID state and its automatically budgeted hot adjacency prefix. DiskANN `warmup_queries` then executes top-1 graph traversal and persisted-vector rerank for each supplied query, priming immutable adjacency and raw-vector LRUs. Other index families treat representative warm-up as resident initialization only.

## Rust

Rust · train, write, and search

    use std::fs::File;

    use paimon_vindex_core::distance::MetricType;
    use paimon_vindex_core::index::{
        VectorIndexConfig, VectorIndexReader, VectorIndexTrainer,
        VectorIndexWriter, VectorSearchParams,
    };
    use paimon_vindex_core::io::PosWriter;

    let config = VectorIndexConfig::IvfSq {
        dimension: 128,
        nlist: 1024,
        metric: MetricType::L2,
    };

    let training = VectorIndexTrainer::train(
        config, &training_vectors, training_count)?;
    let mut writer = VectorIndexWriter::new(training);
    writer.add_vectors(&row_ids, &vectors, vector_count)?;

    let mut file = File::create("vectors.pvindex")?;
    let mut out = PosWriter::new(&mut file);
    writer.write(&mut out)?;

    let file = File::open("vectors.pvindex")?;
    let mut reader = VectorIndexReader::open(file)?;
    reader.optimize_for_search()?;
    let params = VectorSearchParams::automatic(10);
    let (ids, distances) = reader.search(&query, params)?;

Rust · other configurations

    VectorIndexConfig::IvfFlat {
        dimension: 128, nlist: 1024, metric: MetricType::L2,
    };
    VectorIndexConfig::ivf_pq(
        128, 1024, MetricType::L2, false,
    )?;
    VectorIndexConfig::IvfRq {
        dimension: 128, nlist: 1024, bits: 4, metric: MetricType::L2,
    };
    VectorIndexConfig::IvfSq {
        dimension: 128, nlist: 1024, metric: MetricType::L2,
    };

The IVF-PQ constructor uses the default relative PQ-code budget and resolves a concrete `m`. In every option-map API, `pq.m` is optional: `pq.code-ratio=0.0625` is the default, and an explicit `pq.m` takes precedence. Metadata and the on-disk header expose the resolved value.

## C FFI

The C ABI builds `libpaimon_vindex_ffi`; cbindgen produces the public header. Status-returning operations return `0` on success and `-1` on failure. Handle-producing functions such as `paimon_vindex_trainer_open()`, `paimon_vindex_trainer_finish()`, `paimon_vindex_writer_open()`, and `paimon_vindex_reader_open()` return a handle on success or `NULL` on failure. The corresponding free functions return `void` and accept `NULL`. After a failure, `paimon_vindex_last_error()` returns the thread-local diagnostic string, or `NULL` if no error has been recorded.

C · complete lifecycle

    #include "paimon_vindex.h"

    const char *keys[] = {"index.type", "dimension", "nlist", "metric"};
    const char *values[] = {"ivf_flat", "128", "1024", "l2"};

    PaimonVindexTrainerHandle *trainer =
        paimon_vindex_trainer_open(keys, values, 4);
    paimon_vindex_trainer_add_training_vectors(
        trainer, training_vectors, training_count);
    PaimonVindexTrainingHandle *training =
        paimon_vindex_trainer_finish(trainer);
    paimon_vindex_trainer_free(trainer);

    PaimonVindexWriterHandle *writer = paimon_vindex_writer_open(training);
    paimon_vindex_training_free(training);
    paimon_vindex_writer_add_vectors(writer, row_ids, vectors, vector_count);
    paimon_vindex_writer_write_index(writer, output_file);
    paimon_vindex_writer_free(writer);

    PaimonVindexReaderHandle *reader = paimon_vindex_reader_open(input_file);
    PaimonVindexMetadata metadata;
    paimon_vindex_reader_metadata(reader, &metadata);
    paimon_vindex_reader_optimize_for_search(reader);

    int64_t ids[10];
    float distances[10];
    PaimonVindexSearchParams params = {
        .top_k = 10,
        .search_width = PAIMON_VINDEX_SEARCH_WIDTH_AUTO,
        .width = 0,
    };
    paimon_vindex_reader_warmup_queries(reader, representative_queries, 8, 0);
    paimon_vindex_reader_search(reader, query, params, ids, distances, 10);
    paimon_vindex_reader_free(reader);

`PaimonVindexOutputFile` and `PaimonVindexInputFile` are callback structures. The input callback receives every positional range in one I/O batch, allowing object-store implementations to issue reads concurrently. DiskANN batch search may also invoke the callback concurrently from separate query workers, so the callback and its context must be thread-safe. The input structure also carries the three optional read-capability hints described above. C callers own result buffers and pass their capacity explicitly. Metadata reports `pq_m`/`pq_bits` for PQ-backed indexes and `rq_bits` for IVF-RQ. IVF-SQ reports `pq_m=0` and `pq_bits=8` to expose its scalar-code width.

## C++ RAII

`include/paimon_vindex.hpp` wraps the C ABI with explicit ownership and automatic cleanup.

C++

    #include "paimon_vindex.hpp"

    std::vector<std::pair<std::string, std::string>> options = {
        {"index.type", "ivf_flat"}, {"dimension", "128"},
        {"nlist", "1024"}, {"metric", "l2"},
    };

    paimon::vindex::Training training =
        paimon::vindex::Trainer::train(
            options, training_vectors.data(), training_count);
    paimon::vindex::Writer writer(std::move(training));
    writer.add_vectors(row_ids.data(), vectors.data(), vector_count);
    writer.write_index(output_file);

    paimon::vindex::Reader reader(input_file);
    auto metadata = reader.metadata();
    reader.optimize_for_search();
    reader.warmup_queries(representative_queries.data(), 8);
    auto result = reader.search(
        query.data(), paimon::vindex::SearchParams::automatic(10));

## Java / JNI

The package is `org.apache.paimon.index.vector`. String options map directly to Paimon table and index properties; Rust parses and validates them when a Trainer is created.

The Maven JAR contains the supported Linux x86-64, Linux aarch64, macOS arm64, and Windows x86-64 JNI libraries. The first native API call selects the current platform, extracts that library to a private temporary file, and loads it automatically. To use a separately built library instead, start the JVM with `-Dpaimon.vindex.native.path=/absolute/path/to/the/library`.

Java · build and search

    Map<String, String> options = new HashMap<>();
    options.put("index.type", "ivf_sq");
    options.put("metric", "l2");

    try (VectorIndexTraining training =
                 VectorIndexTrainer.train(options, trainingVectors, trainingCount);
         VectorIndexWriter writer = new VectorIndexWriter(training)) {
        writer.addVectors(rowIds, vectors, vectorCount);
        writer.writeIndex(vectorIndexOutput);
    }

    try (VectorIndexReader reader = new VectorIndexReader(vectorIndexInput)) {
        VectorIndexMetadata metadata = reader.metadata();
        reader.optimizeForSearch();
        VectorSearchResult result = reader.search(
                query, VectorSearchParams.automatic(10));
    }

### Batching a large training set

Java · Trainer-owned native staging

    try (VectorIndexTrainer trainer = VectorIndexTrainer.create(options)) {
        for (float[] batch : trainingBatches) {
            trainer.addTrainingVectors(batch, batch.length / dimension);
        }
        try (VectorIndexTraining training = trainer.finishTraining();
             VectorIndexWriter writer = new VectorIndexWriter(training)) {
            writer.addVectors(rowIds, vectors, vectorCount);
            writer.writeIndex(vectorIndexOutput);
        }
    }

Staging avoids one very large Java `float[]` and its array-length limit. Native reservoir sampling bounds retained training rows, but streaming creation still needs concrete `dimension` and either concrete `nlist` or `expected-vector-count`.

## Python

The C++, Java, and Python Readers serialize operations on one native handle across threads. Calling another method on that same handle from its input/output callback is rejected as reentrant—even when DiskANN invokes the callback from a native query worker—instead of deadlocking or re-entering a mutable native handle; use a separate handle if a callback needs vector-index work. The Python package loads `libpaimon_vindex_ffi` with ctypes. A single search returns one-dimensional NumPy arrays. `search_batch` accepts a two-dimensional query array and returns arrays shaped `(query_count, top_k)`.

Python

    from paimon_vindex import (
        SearchParams, VectorIndexReader,
        VectorIndexTrainer, VectorIndexWriter,
    )

    class VectorIndexInput:
        def __init__(self, data: bytes):
            self.data = data
            self.estimated_random_read_latency_nanos = 20_000_000
            self.preferred_window_bytes = 64 * 1024
            self.max_ranges_per_read = 32

        def pread_many(self, ranges):
            return [self.data[pos : pos + length] for pos, length in ranges]

    options = {"index.type": "ivf_sq", "metric": "l2"}
    training = VectorIndexTrainer.train(options, training_vectors)
    writer = VectorIndexWriter(training)
    writer.add_vectors(row_ids, vectors)
    writer.write(output)

    reader = VectorIndexReader(VectorIndexInput(index_bytes))
    reader.optimize_for_search()
    reader.warmup_queries(representative_queries)
    ids, distances = reader.search(
        query, SearchParams.automatic(top_k=10))

## Metadata filter pushdown

The query layer can evaluate metadata predicates through Paimon table or scalar indexes, serialize the allowed row IDs as a 64-bit Roaring bitmap, and pass that set into ANN search as a prefilter. Every binding uses the same wire format:

Language | Single / batch entry points
--- | ---
Rust | `search_with_roaring_filter` / `search_batch_with_roaring_filter`
C | `paimon_vindex_reader_search_with_roaring_filter` / `...search_batch_with_roaring_filter`
Java | `VectorIndexReader.search(..., byte[])` / `searchBatch(..., byte[])`
Python | `search(..., filter_bytes=...)` / `search_batch(..., filter_bytes=...)`

Row-ID constraint`RoaringTreemap` uses the `u64` domain, so row IDs must be non-negative to match the filter. Filters are query payloads and are never written into index files.

DiskANN maps sparse filters through its persisted row-ID lookup after first use; dense filters scan resident row IDs once per batch. It normally scans PQ codes only for matching nodes, then exactly reranks a bounded candidate set. A quality-gated broad filter may use ordinary graph navigation and post-filter its candidates, with a complete matching-node PQ fallback before raw-vector reads. Benchmark filter-heavy traffic separately because the safe PQ path remains linear in the number of matching nodes.