Package {pairwiseLLM}

Title:	Pairwise Comparison Tools for Large Language Model-Based Writing Evaluation
Version:	1.3.1
Description:	Provides a unified framework for generating, submitting, and analyzing pairwise comparisons of writing quality using large language models (LLMs). The package supports live and/or batch evaluation workflows across multiple providers ('OpenAI', 'Anthropic', 'Google Gemini', 'Together AI', and locally-hosted 'Ollama' models), includes bias-tested prompt templates and a flexible template registry, and offers tools for constructing forward and reversed comparison sets to analyze consistency and positional bias. The package additionally supports adaptive pairing workflows that iteratively select comparisons based on model uncertainty to improve ranking efficiency. Results can be modeled using frequentist or Bayesian Bradley–Terry–Luce models (Bradley & Terry, 1952 <doi:10.2307/2334029>; see also Caron & Doucet, 2012 <doi:10.1080/10618600.2012.638220>) or Elo rating methods (see Clark et al., 2018 <doi:10.1371/journal.pone.0190393>) to derive writing quality scores. For information on pairwise comparisons and comparative judgement, see Thurstone (1927) <doi:10.1037/h0070288> and Heldsinger & Humphry (2010) <doi:10.1007/BF03216919>.
License:	MIT + file LICENSE
Encoding:	UTF-8
Imports:	cli, curl, dplyr, httr2, jsonlite, rlang, stats, tibble, tidyselect, tools, utils
Suggests:	BradleyTerry2, cmdstanr, EloChoice, future, future.apply, knitr, mockery, pkgload, purrr, readr, rmarkdown, sirt, stringr, testthat (≥ 3.0.0), tidyr, vctrs, withr
Additional_repositories:	https://stan-dev.r-universe.dev/
SystemRequirements:	CmdStan and a C++17 toolchain (optional; needed only for Bayesian and adaptive modeling)
Config/testthat/edition:	3
URL:	https://github.com/shmercer/pairwiseLLM, https://shmercer.github.io/pairwiseLLM/
BugReports:	https://github.com/shmercer/pairwiseLLM/issues
Depends:	R (≥ 4.1)
VignetteBuilder:	knitr
Collate:	'adaptive_benchmark_metrics.R' 'adaptive_btl_refit.R' 'adaptive_candidates.R' 'adaptive_constraints.R' 'adaptive_linking_calibration.R' 'adaptive_linking_phase_a.R' 'adaptive_logs.R' 'adaptive_persist.R' 'adaptive_print.R' 'adaptive_rank.R' 'adaptive_round_candidates.R' 'adaptive_run.R' 'adaptive_schemas.R' 'adaptive_select.R' 'adaptive_simulation_harness.R' 'adaptive_state.R' 'adaptive_step.R' 'adaptive_trueskill.R' 'adaptive_utility.R' 'anthropic_batch_api.R' 'anthropic_live.R' 'api_keys.R' 'bayes_btl_mcmc.R' 'bayes_btl_mcmc_adaptive.R' 'bayes_btl_summarize.R' 'bt_helpers.R' 'bt_model.R' 'btl_mcmc_constraints.R' 'btl_mcmc_contracts.R' 'btl_mcmc_fit_contracts.R' 'btl_mcmc_ingest.R' 'btl_mcmc_model_variant.R' 'btl_mcmc_state.R' 'btl_mcmc_stopping.R' 'btl_mcmc_summaries.R' 'btl_mcmc_theta_summary_stubs.R' 'core_budget.R' 'cost_estimator.R' 'custom_id.R' 'data-example_writing.R' 'data_import.R' 'draws_sanitize.R' 'elo_model.R' 'gemini_batch_api.R' 'gemini_live.R' 'gemini_params.R' 'httr2_retry.R' 'llm_backends.R' 'llm_batch.R' 'llm_multi_batch.R' 'normalize_results.R' 'ollama_live.R' 'openai_batch_api.R' 'openai_live.R' 'openai_params.R' 'pairing.R' 'pairwiseLLM-package.R' 'prompt_template.R' 'reverse_consistency.R' 'seed_helpers.R' 'together_live.R' 'traits.R' 'utils-null-coalesce.R' 'vertex_live.R'
Config/roxygen2/version:	8.1.0
NeedsCompilation:	no
Packaged:	2026-09-06 23:51:09 UTC; sterett
Author:	Sterett H. Mercer [aut, cre, cph]
Maintainer:	Sterett H. Mercer <sterett.mercer@ubc.ca>
Repository:	CRAN
Date/Publication:	2026-09-07 06:40:11 UTC

pairwiseLLM: Pairwise comparisons and adaptive ranking with LLM judges

Description

pairwiseLLM supports (1) constructing paired-comparison designs over items, (2) collecting binary judgments from LLMs (live or via provider batch APIs), and (3) fitting ranking models (Elo, Bradley–Terry) including an adaptive, Bayesian BTL (MCMC) workflow.

Details

Typical workflow (most users)

1. Load items using read_samples_df() or read_samples_dir().

2. Create a pairing design with make_pairs() or sample_pairs(), optionally adding reversals via sample_reverse_pairs().

3. (Optional) Control ordering / bias with randomize_pair_order(), alternate_pair_order(), and diagnostics like check_positional_bias() or reversal agreement via compute_reverse_consistency().

4. Build prompts with build_prompt() + trait_description(), using templates managed by list_prompt_templates(), get_prompt_template(), set_prompt_template(), register_prompt_template(), and remove_prompt_template().

5. Collect judgments:

   • Live: submit_llm_pairs() or provider wrappers like submit_openai_pairs_live(), submit_anthropic_pairs_live(), submit_gemini_pairs_live(), submit_vertex_pairs_live(), submit_ollama_pairs_live(), submit_together_pairs_live().

   • Batch (recommended for scale): run_openai_batch_pipeline(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(). Vertex is currently supported on the live path only.

6. Assemble modeling data with build_bt_data() or build_elo_data() (and for some pipelines, build_btl_results_data()).

7. Fit / summarize:

   • Elo: fit_elo_model()

   • Bradley–Terry: fit_bt_model()

   • Bayesian BTL (MCMC): fit_bayes_btl_mcmc()

   • Summaries: summarize_bt_fit(), summarize_items(), summarize_refits()

Provider matrix

• Live backends: "openai", "anthropic", "gemini" (Gemini Developer API), "vertex" (Vertex AI Gemini API), "together", and "ollama".

• Batch backends: "openai", "anthropic", and "gemini". backend = "vertex" is rejected explicitly on generic batch surfaces.

• Gemini Developer API and Vertex use separate API-key surfaces: GEMINI_API_KEY and VERTEX_API_KEY.

• service_tier is provider-specific. Gemini Developer API and Vertex share the public values "standard", "flex", and "priority", but they use different request transports.

Adaptive Bayesian pairing + ranking (end-to-end loop)

If you want the package to both choose pairs and fit Bayesian BTL in an auditable loop, use the adaptive workflow:

• Start a session with adaptive_rank_start() (or use the wrapper adaptive_rank()).

• Run live rounds with adaptive_rank_run_live().

• Resume an existing run with adaptive_rank_resume().

• Persist / reload sessions with save_adaptive_session(), load_adaptive_session(), and validate directories with validate_session_dir().

Exported functions by task

1) Read / define items to rank

• read_samples_df(), read_samples_dir()

2) Create pair designs and manage ordering

• Pair construction: make_pairs(), sample_pairs(), sample_reverse_pairs()

• Order helpers: randomize_pair_order(), alternate_pair_order()

• Consistency / bias: compute_reverse_consistency(), check_positional_bias()

3) Prompts and template registry

• Prompt building: build_prompt(), trait_description()

• Templates: list_prompt_templates(), get_prompt_template(), set_prompt_template(), register_prompt_template(), remove_prompt_template()

4) LLM judging (provider-agnostic core)

• Utilities: check_llm_api_keys(), estimate_llm_pairs_cost()

• Live submission: submit_llm_pairs(), llm_compare_pair()

• Batch orchestration (generic): llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), llm_download_batch_results(), llm_resume_multi_batches()

5) Provider-specific: live helpers

• OpenAI: submit_openai_pairs_live(), openai_compare_pair_live()

• Anthropic: submit_anthropic_pairs_live(), anthropic_compare_pair_live()

• Gemini: submit_gemini_pairs_live(), gemini_compare_pair_live()

• Vertex: submit_vertex_pairs_live(), vertex_compare_pair_live()

• Ollama: submit_ollama_pairs_live(), ollama_compare_pair_live(), ensure_only_ollama_model_loaded()

• Together: submit_together_pairs_live(), together_compare_pair_live()

6) Provider-specific: batch pipelines + low-level batch helpers

• OpenAI pipeline: run_openai_batch_pipeline()

  • Build / write / upload: build_openai_batch_requests(), write_openai_batch_file(), openai_upload_batch_file()

  • Manage jobs: openai_create_batch(), openai_get_batch(), openai_poll_batch_until_complete()

  • Download / parse: openai_download_batch_output(), parse_openai_batch_output()

• Anthropic pipeline: run_anthropic_batch_pipeline()

  • Build: build_anthropic_batch_requests()

  • Manage jobs: anthropic_create_batch(), anthropic_get_batch(), anthropic_poll_batch_until_complete()

  • Download / parse: anthropic_download_batch_results(), parse_anthropic_batch_output()

• Gemini pipeline: run_gemini_batch_pipeline()

  • Build: build_gemini_batch_requests()

  • Manage jobs: gemini_create_batch(), gemini_get_batch(), gemini_poll_batch_until_complete()

  • Download / parse: gemini_download_batch_results(), parse_gemini_batch_output()

7) Modeling: build data, fit models, summarize

• Data builders: build_bt_data(), build_elo_data(), build_btl_results_data()

• Fits: fit_bt_model(), fit_elo_model(), fit_bayes_btl_mcmc()

• Summaries: summarize_bt_fit(), summarize_items(), summarize_refits()

8) Adaptive workflow: run control, persistence, and summaries

• Control: adaptive_rank(), adaptive_rank_start(), adaptive_rank_run_live(), adaptive_rank_resume()

• Session IO: save_adaptive_session(), load_adaptive_session(), validate_session_dir()

• Adaptive summaries: summarize_adaptive()

• Logging accessors (for audit / analysis): adaptive_get_logs(), adaptive_round_log(), adaptive_step_log(), adaptive_item_log(), adaptive_results_history()

• Advanced: make_adaptive_judge_llm()

Design notes (adaptive Bayesian BTL)

The adaptive design targets stable, auditable rankings using Bayesian Bradley–Terry–Luce inference with MCMC, supports position bias and lapse-rate variants, and is intended to be robust for noisy LLM judges.

The adaptive loop enforces connectivity, duplicate control with order reversal for repeated unordered pairs, and approximate 50/50 position balance across items. Stopping and refits are logged so stop decisions are reproducible from the round_log fields (an explicit stop audit trail).

Author(s)

Maintainer: Sterett H. Mercer sterett.mercer@ubc.ca (ORCID) [copyright holder]

Authors:

• Sterett H. Mercer sterett.mercer@ubc.ca (ORCID) [copyright holder]

See Also

• For end-to-end adaptive ranking: adaptive_rank_run_live() or adaptive_rank().

• For large-scale provider batching: run_openai_batch_pipeline().

Internal: Google Gemini API key helper

Description

This is a thin wrapper around .get_api_key() for the Google Gemini backend. It looks for a GEMINI_API_KEY environment variable by default and can be overridden explicitly via the api_key argument.

Usage

.gemini_api_key(api_key = NULL)

Arguments

Internal: parse a Gemini GenerateContentResponse into the standard tibble row

Description

For batch responses, Gemini 3 Pro currently typically returns:

• candidates[[1]]$content$parts[[1]]$text = final answer

• candidates[[1]]$content$parts[[1]]$thoughtSignature = opaque signature

• usageMetadata$thoughtsTokenCount = hidden reasoning tokens

Usage

.parse_gemini_pair_response(
  custom_id,
  ID1,
  ID2,
  response,
  include_thoughts = FALSE
)

Details

When include_thoughts = TRUE and >= 2 parts are present, we mirror the live behavior: first part = thoughts, remaining parts = content. When only one part is present, we treat it as content and leave thoughts as NA (batch isn't returning visible thoughts text).

Internal: Together.ai API key helper

Description

This is a thin wrapper around .get_api_key() for the Together.ai backend. It looks for a TOGETHER_API_KEY environment variable by default and can be overridden explicitly via the api_key argument.

Usage

.together_api_key(api_key = NULL)

Arguments

Internal: Vertex AI Gemini API key helper

Description

This is a thin wrapper around .get_api_key() for the Vertex AI Gemini backend. It looks for a VERTEX_API_KEY environment variable by default and can be overridden explicitly via the api_key argument.

Usage

.vertex_api_key(api_key = NULL)

Arguments

Retrieve canonical adaptive logs.

Description

Retrieve canonical adaptive logs.

Usage

adaptive_get_logs(state)

Arguments

Details

Returns the canonical adaptive logs as currently held in memory: step_log, round_log, item_log, and link_stage_log. These correspond to step attempts, posterior refit rounds, item-level refit summaries, and per-refit linking summaries.

Value

A named list with four elements:

step_log

A tibble with one row per attempted step.

round_log

A tibble with one row per BTL refit round.

item_log

A list of per-refit item tibbles.

link_stage_log

A tibble with one row per (refit_id, spoke_id) linking summary when linking mode is active.

See Also

adaptive_step_log(), adaptive_round_log(), adaptive_item_log()

Other adaptive logs: adaptive_item_log(), adaptive_results_history(), adaptive_round_log(), adaptive_step_log(), summarize_items(), summarize_refits()

Examples

state <- adaptive_rank_start(c("a", "b", "c"), seed = 1)
logs <- adaptive_get_logs(state)
names(logs)

Adaptive item log accessor.

Description

Adaptive item log accessor.

Usage

adaptive_item_log(state, refit_id = NULL, stack = FALSE)

Arguments

Details

item_log stores per-item posterior summaries by refit. The underlying state stores a list of refit tables; this accessor can return one refit table (default: most recent) or stack all refits into a single tibble.

Item-level summaries are domain-explicit:

• theta_raw_*: raw/within-set posterior summaries (EAP, fixed quantiles, SD, rank) at the current refit.

• theta_link_*: linked/global posterior summaries (EAP, fixed quantiles, SD, rank) after transform application.

• During linking Phase A (phase_scope = "phase_a_set"), theta_link_* is typed NA by design.

• phase_scope, phase_scope_set_id, and in_phase_scope indicate which item domain is currently optimized.

Value

A tibble of item-level summaries. When stack = FALSE, one row per item for the selected refit. When stack = TRUE, one row per item per refit with refit_id identifying source refit.

See Also

adaptive_get_logs(), summarize_items(), adaptive_round_log()

Other adaptive logs: adaptive_get_logs(), adaptive_results_history(), adaptive_round_log(), adaptive_step_log(), summarize_items(), summarize_refits()

Examples

state <- adaptive_rank_start(c("a", "b", "c"), seed = 1)
adaptive_item_log(state)

Run adaptive ranking end-to-end from data and model settings

Description

High-level workflow wrapper that reads sample data, constructs an LLM judge, starts or resumes adaptive state, runs adaptive_rank_run_live(), and returns state plus summary outputs.

Usage

adaptive_rank(
  data,
  id_col = 1,
  text_col = 2,
  backend = c("openai", "anthropic", "gemini", "vertex", "together", "ollama"),
  model = NULL,
  trait = "overall_quality",
  trait_name = NULL,
  trait_description = NULL,
  prompt_template = set_prompt_template(),
  endpoint = "chat.completions",
  api_key = NULL,
  include_raw = FALSE,
  judge_args = list(),
  judge_call_args = list(),
  n_steps = 1L,
  fit_fn = NULL,
  adaptive_config = NULL,
  btl_config = NULL,
  session_dir = NULL,
  persist_item_log = FALSE,
  checkpoint_every_steps = NULL,
  resume = TRUE,
  seed = 1L,
  progress = c("all", "refits", "steps", "none"),
  progress_redraw_every = 10L,
  progress_show_events = TRUE,
  progress_errors = TRUE,
  save_outputs = FALSE,
  output_file = NULL,
  judge = NULL
)

Arguments

Details

This helper is designed for end users who want one entry point for adaptive runs. It supports:

• data input from a data frame, file (.csv, .tsv, .txt, .rds), or a directory of .txt files;

• model/backend configuration through make_adaptive_judge_llm();

• all adaptive runtime controls exposed by adaptive_rank_run_live();

• wrapper-visible phase_a reuse surfaces (manifest, artifact_dir, and per-set status) for separate-run then later-link workflows;

• resumability via session_dir and resume;

• optional saving of run outputs to an .rds artifact.

Model options: use judge_args (fixed) and judge_call_args (per-run overrides) to pass any additional llm_compare_pair() arguments, including provider-specific controls such as reasoning, service_tier, temperature, top_p, logprobs, include_thoughts, or host.

Adaptive options: all key controls from adaptive_rank_run_live() are available directly: n_steps, fit_fn, adaptive_config, btl_config, progress, progress_redraw_every, progress_show_events, progress_errors, session_dir, and persist_item_log. Use adaptive_config for identifiability-gated controller behavior and btl_config for inference/diagnostics cadence only.

Linking run modes: run_mode = "within_set" is the single-set workflow. run_mode = "link_one_spoke" and run_mode = "link_multi_spoke" require multi-set input (set_id/global_item_id), enforce hub<->spoke routing defaults, and preserve Phase A artifact gating before Phase B cross-set comparisons begin. Phase B uses anchored-joint estimation with a hard-locked hub and global-shared judge parameters. Every wrapper call returns canonical phase_a outputs that can be fed back into a later linking run through adaptive_config$phase_a_artifacts.

Selection semantics: pair selection is TrueSkill-driven in one-pair transactional steps. Rolling anchors are refreshed from current score proxies and anchor-link routing compares exactly one anchor endpoint with one non-anchor endpoint. Long/mid-link routing excludes anchor-anchor and anchor-non-anchor pairs, while local-link routing admits same-stratum pairs and anchor-involving pairs according to stage bounds.

Wrapper-visible defaults include top-band refinement (top_band_pct = 0.10, top_band_bins = 5) with top-band size computed as ceiling(top_band_pct * N).

Exposure and repeat routing: under-represented routing is degree-based (deg <= D_min + 1), while repeat-pressure gating is based on recent exposure (bottom-quantile recent_deg with quantile default 0.25) and per-endpoint repeat slot accounting.

Inference separation: BTL refits are used for posterior inference, diagnostics, stop logic, and the long-link posterior gate after an accepted refit is available. They are not used to choose the next pair.

Resume behavior: when resume = TRUE and session_dir already contains adaptive artifacts, failed session loads abort with an actionable error instead of starting a fresh run silently.

Value

A list with:

state

Final adaptive_state.

summary

Run-level summary from summarize_adaptive().

refits

Per-refit summary from summarize_refits().

items

Item summary from summarize_items(), sorted by a usable canonical rank column (rank_link for linking runs when available, otherwise rank_raw).

logs

Canonical logs from adaptive_get_logs().

phase_a

Canonical wrapper-visible Phase A discovery surface with per-set status, artifact_dir, artifact_paths, and a reusable manifest that can be fed back into a later linking run via adaptive_config$phase_a_artifacts.

output_file

Saved output path when save_outputs = TRUE, otherwise NULL.

See Also

make_adaptive_judge_llm(), adaptive_rank_run_live(), adaptive_rank_start(), adaptive_rank_resume(), llm_compare_pair()

Other adaptive ranking: adaptive_rank_resume(), adaptive_rank_run_live(), adaptive_rank_start(), make_adaptive_judge_llm(), summarize_adaptive()

Examples

data("example_writing_samples", package = "pairwiseLLM")

out <- adaptive_rank(
  data = example_writing_samples[1:8, c("ID", "text", "quality_score")],
  id_col = "ID",
  text_col = "text",
  model = "gpt-5.1",
  judge = function(A, B, state, ...) {
    y <- as.integer(A$quality_score[[1]] >= B$quality_score[[1]])
    list(is_valid = TRUE, Y = y, invalid_reason = NA_character_)
  },
  n_steps = 4,
  progress = "none"
)

out$summary
head(out$logs$step_log)

## Not run: 
# Live run with OpenAI gpt-5.1 + lower-cost Flex processing.
live <- adaptive_rank(
  data = example_writing_samples[1:12, c("ID", "text")],
  backend = "openai",
  model = "gpt-5.1",
  endpoint = "responses",
  judge_args = list(
    reasoning = "low",
    service_tier = "flex",
    include_thoughts = FALSE
  ),
  btl_config = list(
    refit_pairs_target = 20L,
    ess_bulk_min = 500,
    eap_reliability_min = 0.92
  ),
  adaptive_config = list(
    explore_taper_mult = 0.40,
    star_override_budget_per_round = 2L
  ),
  n_steps = 120,
  session_dir = file.path(tempdir(), "adaptive-live"),
  persist_item_log = TRUE,
  resume = TRUE,
  progress = "all",
  save_outputs = TRUE
)

print(live$state)
live$summary

# Wrapper-driven linking workflow (hub + one spoke).
linking_samples <- example_writing_samples[1:12, c("ID", "text")]
linking_samples$set_id <- rep(c(1L, 2L), each = 6L)
linking_samples$global_item_id <- paste0("g_", linking_samples$ID)

link_out <- adaptive_rank(
  data = linking_samples,
  id_col = "ID",
  text_col = "text",
  backend = "openai",
  model = "gpt-5.1",
  adaptive_config = list(
    run_mode = "link_one_spoke",
    hub_id = 1L,
    phase_a_mode = "run",
    probe_panel_edges = 48L,
    hub_anchor_required_phase_b = TRUE,
    max_pairs_after_stop = 0L
  ),
  n_steps = 200,
  session_dir = file.path(tempdir(), "adaptive-link"),
  resume = TRUE,
  progress = "refits"
)

# Later linking from prior wrapper outputs:
# hub_run <- adaptive_rank(
#   data = linking_samples[linking_samples$set_id == 1L, c("ID", "text")],
#   backend = "openai",
#   model = "gpt-5.1",
#   n_steps = 120,
#   progress = "none"
# )
# spoke_run <- adaptive_rank(
#   data = linking_samples[linking_samples$set_id == 2L, c("ID", "text")],
#   backend = "openai",
#   model = "gpt-5.1",
#   n_steps = 120,
#   progress = "none"
# )
#
# link_out <- adaptive_rank(
#   data = linking_samples,
#   id_col = "ID",
#   text_col = "text",
#   backend = "openai",
#   model = "gpt-5.1",
#   adaptive_config = list(
#     run_mode = "link_one_spoke",
#     hub_id = 1L,
#     phase_a_mode = "import",
#     phase_a_artifacts = list(
#       `1` = hub_run$phase_a$manifest,
#       `2` = spoke_run$phase_a$artifact_dir
#     )
#   ),
#   n_steps = 200,
#   progress = "refits"
# )

names(link_out$logs)

## End(Not run)

Adaptive ranking resume

Description

Resume a previously persisted adaptive pairing session.

Usage

adaptive_rank_resume(session_dir, ...)

Arguments

Details

This is a thin wrapper around load_adaptive_session() and performs schema and log-shape checks during load. Returned state preserves canonical step_log, round_log, and item_log contents used for adaptive auditability.

Value

An adaptive_state object restored from disk.

See Also

adaptive_rank_start(), adaptive_rank_run_live(), save_adaptive_session(), load_adaptive_session()

Other adaptive ranking: adaptive_rank(), adaptive_rank_run_live(), adaptive_rank_start(), make_adaptive_judge_llm(), summarize_adaptive()

Examples

dir <- tempfile("pwllm-session-")
state <- adaptive_rank_start(c("a", "b", "c"), seed = 3)
save_adaptive_session(state, dir, overwrite = TRUE)
restored <- adaptive_rank_resume(dir)
summarize_adaptive(restored)

Adaptive ranking live runner

Description

Execute stepwise adaptive ranking with a user-supplied judge.

Usage

adaptive_rank_run_live(
  state,
  judge,
  n_steps = 1L,
  fit_fn = NULL,
  adaptive_config = NULL,
  btl_config = NULL,
  session_dir = NULL,
  persist_item_log = NULL,
  checkpoint_every_steps = NULL,
  progress = c("all", "refits", "steps", "none"),
  progress_redraw_every = 10L,
  progress_show_events = TRUE,
  progress_errors = TRUE,
  ...
)

Arguments

Details

Each iteration attempts at most one pair evaluation ("one-pair step"), then applies transactional updates if and only if the judge response is valid. Invalid responses produce a logged step with pair_id = NA and must not update committed-comparison state.

Within-set routing is TrueSkill-based with utility

U_0 = p_{ij}(1 - p_{ij})

. After an accepted posterior refit is available, the long-link gate uses the BTL posterior win probability for candidate eligibility; before that it falls back deterministically to TrueSkill. In linking Phase B, anchor/strata routing uses linking-global scores built from Phase A summaries and the accepted anchored-joint state. Linking Phase B routing ranks eligible cross-set candidates by ridge-stabilized D-optimal log-det information gain on the active linking parameter block using order-averaged Model D probabilities. Linking inference parameters remain inference-only (diagnostics and stopping) and are not direct pair-selection objectives. Phase B uses a hard-lock hub-fixed fit and a deterministic accepted state before the first linking refit. Exploration/exploitation routing and fallback handling are recorded in step_log.

Round scheduling uses stage-specific admissibility:

• rolling-anchor links compare one anchor and one non-anchor endpoint;

• long/mid links exclude anchor endpoints and enforce stratum-distance bounds;

• local-link routing admits same-stratum pairs and anchor-involving pairs within local stage bounds.

Exposure and repeat handling are soft, stage-local constraints: under-represented exploration uses degree set deg <= D_min + 1, while repeat-pressure gating uses bottom-quantile recent_deg (default quantile 0.25) and per-endpoint repeat-slot accounting against repeat_in_round_budget.

Top-band defaults for stratum construction are top_band_pct = 0.10 and top_band_bins = 5, with top-band size ceiling(top_band_pct * N).

Bayesian BTL refits are triggered on step-based cadence and evaluated with diagnostics gates (including ESS thresholds), reliability, and lagged stability criteria. Refit-level outcomes are appended to round_log; per-item posterior summaries are appended to item_log. Controller behavior can change after refits via identifiability-gated settings in adaptive_config; those controls affect pair routing and quotas, while BTL remains inference-only. If adaptive_config$max_pairs_after_stop > 0, the run records a stop boundary at the first refit with stop_decision = TRUE and allows at most that many additional committed comparisons before deterministic termination. Round logs record max_pairs_after_stop and pairs_committed_after_stop.

Value

An updated adaptive_state. The returned state includes appended step_log rows for attempted steps and, when refits occur, appended round_log and item_log entries.

See Also

adaptive_rank_start(), adaptive_rank_resume(), adaptive_step_log(), adaptive_round_log(), adaptive_item_log()

Other adaptive ranking: adaptive_rank(), adaptive_rank_resume(), adaptive_rank_start(), make_adaptive_judge_llm(), summarize_adaptive()

Examples

# ------------------------------------------------------------------
# Offline end-to-end workflow (fast, deterministic, CRAN-safe)
# ------------------------------------------------------------------
data("example_writing_samples", package = "pairwiseLLM")

items <- dplyr::rename(
  example_writing_samples[1:8, c("ID", "text", "quality_score")],
  item_id = ID
)

# Use the package defaults for trait and prompt template.
trait <- trait_description("overall_quality")
prompt_template <- set_prompt_template()

# Deterministic local judge based on fixture quality scores.
sim_judge <- function(A, B, state, ...) {
  y <- as.integer(A$quality_score[[1]] >= B$quality_score[[1]])
  list(is_valid = TRUE, Y = y, invalid_reason = NA_character_)
}

session_dir <- tempfile("pwllm-adaptive-session-")

state <- adaptive_rank_start(
  items = items,
  seed = 42,
  adaptive_config = list(
    global_identified_reliability_min = 0.85,
    star_override_budget_per_round = 2L
  ),
  session_dir = session_dir,
  persist_item_log = TRUE
)

state <- adaptive_rank_run_live(
  state = state,
  judge = sim_judge,
  n_steps = 6,
  btl_config = list(
    # Keep examples lightweight while showing custom stop config inputs.
    refit_pairs_target = 50L,
    ess_bulk_min = 400,
    eap_reliability_min = 0.90
  ),
  adaptive_config = list(
    explore_taper_mult = 0.40,
    boundary_frac = 0.20
  ),
  progress = "steps",
  progress_redraw_every = 1L,
  progress_show_events = TRUE,
  progress_errors = TRUE
)

# Print and inspect run outputs.
print(state)
run_summary <- summarize_adaptive(state)
step_view <- adaptive_step_log(state)
logs <- adaptive_get_logs(state)

run_summary
head(step_view)
names(logs)

# Resume from disk and continue.
resumed <- adaptive_rank_resume(session_dir)
resumed <- adaptive_rank_run_live(
  state = resumed,
  judge = sim_judge,
  n_steps = 4,
  progress = "none"
)
summarize_adaptive(resumed)

# ------------------------------------------------------------------
# Live OpenAI workflow via backend-agnostic llm_compare_pair()
# ------------------------------------------------------------------
## Not run: 
# Requires network + OPENAI_API_KEY. This incurs API cost.
# check_llm_api_keys() is a quick preflight.
check_llm_api_keys()

data("example_writing_samples", package = "pairwiseLLM")
live_items <- dplyr::rename(
  example_writing_samples[1:12, c("ID", "text")],
  item_id = ID
)

# Default trait/template setup used by the backend-agnostic runner.
trait <- trait_description("overall_quality")
prompt_template <- set_prompt_template()

live_session_dir <- file.path(tempdir(), "pwllm-adaptive-openai")

judge_openai <- function(A, B, state, ...) {
  res <- llm_compare_pair(
    ID1 = A$item_id[[1]],
    text1 = A$text[[1]],
    ID2 = B$item_id[[1]],
    text2 = B$text[[1]],
    model = "gpt-5.1",
    trait_name = trait$name,
    trait_description = trait$description,
    prompt_template = prompt_template,
    backend = "openai",
    endpoint = "responses",
    reasoning = "low",
    service_tier = "flex",
    include_thoughts = FALSE,
    temperature = NULL,
    top_p = NULL,
    logprobs = NULL
  )

  better_id <- res$better_id[[1]]
  ok_ids <- c(A$item_id[[1]], B$item_id[[1]])
  if (is.na(better_id) || !(better_id %in% ok_ids)) {
    return(list(
      is_valid = FALSE,
      Y = NA_integer_,
      invalid_reason = "model_response_invalid"
    ))
  }

  list(
    is_valid = TRUE,
    Y = as.integer(identical(better_id, A$item_id[[1]])),
    invalid_reason = NA_character_
  )
}

state_live <- adaptive_rank_start(
  items = live_items,
  seed = 2026,
  session_dir = live_session_dir,
  persist_item_log = TRUE
)

state_live <- adaptive_rank_run_live(
  state = state_live,
  judge = judge_openai,
  n_steps = 120L,
  btl_config = list(
    refit_pairs_target = 20L,
    ess_bulk_min = 500,
    ess_bulk_min_near_stop = 1200,
    max_rhat = 1.01,
    divergences_max = 0L,
    eap_reliability_min = 0.92,
    stability_lag = 2L,
    theta_corr_min = 0.97,
    theta_sd_rel_change_max = 0.08,
    rank_spearman_min = 0.97
  ),
  progress = "all",
  progress_redraw_every = 1L,
  progress_show_events = TRUE,
  progress_errors = TRUE
)

# Reporting outputs for end users.
print(state_live)
run_summary <- summarize_adaptive(state_live)
refit_summary <- summarize_refits(state_live)
item_summary <- summarize_items(state_live)
logs <- adaptive_get_logs(state_live)

# Store outputs for audit/reproducibility.
saveRDS(
  list(
    run_summary = run_summary,
    refit_summary = refit_summary,
    item_summary = item_summary,
    logs = logs
  ),
  file.path(live_session_dir, "adaptive_outputs.rds")
)

# Resume from stored state and continue sampling.
state_live <- adaptive_rank_resume(live_session_dir)
state_live <- adaptive_rank_run_live(
  state = state_live,
  judge = judge_openai,
  n_steps = 40L,
  progress = "refits"
)
print(summarize_adaptive(state_live))

## End(Not run)

Adaptive ranking

Description

Initialize an adaptive ranking session and canonical state object.

Usage

adaptive_rank_start(
  items,
  seed = 1L,
  session_dir = NULL,
  persist_item_log = FALSE,
  ...,
  adaptive_config = NULL,
  checkpoint_every_steps = NULL
)

Arguments

Details

This function creates the stepwise controller state and seeds all canonical logs used in the adaptive pairing workflow. Warm start pair construction follows the shuffled chain design, which guarantees a connected comparison graph after N - 1 committed comparisons.

Pair selection in this framework is stepwise and uncertainty-aware. Within-set routing uses TrueSkill base utility

U_0 = p_{ij}(1 - p_{ij})

where p_{ij} is the current TrueSkill win probability for pair \{i, j\}. In linking Phase B, anchor/strata routing uses a linking-global score derived from Phase A raw summaries and the accepted Phase B linking state. In linking Phase B, eligible cross-set candidates are ranked by ridge-stabilized D-optimal log-det information gain on the active linking parameter block using order-averaged Model D probabilities. In the spoke free block with the hub fixed. Linking inference parameters are used for inference/diagnostics/stopping, not as direct selection objectives. Phase B uses pooled within-set Phase A judge-parameter estimates, using the configured BTL model variant, as the accepted shared source for fixed beta/epsilon constants. Bayesian BTL posterior draws are not used as general pair-selection objectives; within-set pairing remains TrueSkill-routed, with accepted posterior refits contributing only to the long-link probability gate. Linking Phase B refits use Bayesian posterior estimation and posterior summaries/diagnostics are logged per spoke at each linking refit.

The returned state contains canonical logs:

• step_log: one row per attempted step,

• round_log: one row per posterior refit,

• item_log: per-item posterior summaries by refit.

If session_dir is supplied, the initialized state is persisted immediately using save_adaptive_session().

Value

An adaptive state object containing step_log, round_log, and item_log. The object includes class "adaptive_state", item ID mappings, TrueSkill state, warm-start queue, refit metadata, and runtime configuration.

See Also

adaptive_rank_run_live(), adaptive_rank_resume(), adaptive_step_log(), adaptive_round_log(), adaptive_item_log()

Other adaptive ranking: adaptive_rank(), adaptive_rank_resume(), adaptive_rank_run_live(), make_adaptive_judge_llm(), summarize_adaptive()

Examples

state <- adaptive_rank_start(c("a", "b", "c"), seed = 11)
summarize_adaptive(state)

Adaptive results history in build_bt_data() format.

Description

Adaptive results history in build_bt_data() format.

Usage

adaptive_results_history(state, committed_only = TRUE)

Arguments

Details

Converts adaptive step outcomes into the three-column format used by build_bt_data() (object1, object2, result). With committed_only = TRUE, only committed steps (pair_id not missing) are retained. This preserves the transactional invariant that invalid steps do not contribute to inferred comparisons.

Value

A tibble with columns:

object1

Character item id shown in position A.

object2

Character item id shown in position B.

result

Numeric outcome in {0, 1} where 1 means object1 wins.

See Also

build_bt_data(), adaptive_step_log()

Other adaptive logs: adaptive_get_logs(), adaptive_item_log(), adaptive_round_log(), adaptive_step_log(), summarize_items(), summarize_refits()

Examples

state <- adaptive_rank_start(c("a", "b", "c"), seed = 1)
adaptive_results_history(state)

Adaptive round log accessor.

Description

Adaptive round log accessor.

Usage

adaptive_round_log(state, reconstruct_deferred = FALSE)

Arguments

Details

round_log is the canonical per-refit audit log for the adaptive pairing workflow. Each row summarizes one Bayesian BTL refit and includes diagnostics, reliability, and stopping-gate fields used to justify stop decisions.

Core columns:

• Refit identity/state: refit_id, round_id_at_refit, step_id_at_refit, timestamp, model_variant, n_items, total_pairs_done, new_pairs_since_last_refit, n_unique_pairs_seen.

• Candidate health: proposed_pairs_mode, starve_rate_since_last_refit, fallback_rate_since_last_refit, fallback_used_mode, starvation_reason_mode.

• Identifiability/quota adaptation: global_identified, global_identified_reliability_min, global_identified_rank_corr_min, long_quota_raw, long_quota_effective, long_quota_removed, realloc_to_mid, realloc_to_local, phase_scope, phase_scope_set_id, phase_scope_n_items.

• Coverage/imbalance: mean_degree, min_degree, mean_degree_scope, min_degree_scope, pos_balance_sd, star_cap_rejects_since_last_refit, star_cap_reject_rate_since_last_refit, recent_deg_median_since_last_refit, recent_deg_max_since_last_refit.

• Posterior parameter summaries: epsilon_mean/percentiles and b_mean/percentiles.

• Audit diagnostics: ts_sigma_mean, ts_sigma_max, ts_degree_sigma_corr, ts_btl_theta_corr, ts_btl_rank_spearman, ci95_theta_width_*, near_tie_adj_frac, near_tie_adj_count, p_adj_median, cov_trace_theta, cov_logdet_diag_theta, post_sd_theta_p10, post_sd_theta_p50, post_sd_theta_p90, top20_boundary_entropy_*, nn_diff_sd_*.

• Stopping diagnostics: diagnostics_pass, diagnostics_divergences_pass, diagnostics_rhat_pass, diagnostics_ess_pass, divergences, divergences_max_allowed, max_rhat, max_rhat_allowed, min_ess_bulk, ess_bulk_required, near_stop_active, reliability_EAP, reliability_EAP_scope, eap_reliability_min, eap_pass, eap_pass_scope, theta_sd_eap, theta_sd_eap_scope, rho_theta, rho_theta_scope, lag_eligible, lag_eligible_scope, theta_corr_min, theta_corr_pass, theta_corr_pass_scope, delta_sd_theta, delta_sd_theta_scope, theta_sd_rel_change_max, delta_sd_theta_pass, delta_sd_theta_pass_scope, rho_rank, rho_rank_scope, rank_spearman_min, rho_rank_pass, rho_rank_pass_scope.

• Refit execution metadata: mcmc_chains, mcmc_parallel_chains, mcmc_core_fraction, mcmc_cores_detected_physical, mcmc_cores_detected_logical, mcmc_threads_per_chain, mcmc_cmdstanr_version.

• Stop output: stop_decision, stop_reason, max_pairs_after_stop, pairs_committed_after_stop.

Value

A tibble with one row per completed posterior refit round.

See Also

adaptive_get_logs(), summarize_refits(), adaptive_rank_run_live()

Other adaptive logs: adaptive_get_logs(), adaptive_item_log(), adaptive_results_history(), adaptive_step_log(), summarize_items(), summarize_refits()

Examples

state <- adaptive_rank_start(c("a", "b", "c"), seed = 1)
adaptive_round_log(state)

Adaptive step log accessor.

Description

Adaptive step log accessor.

Usage

adaptive_step_log(state)

Arguments

Details

step_log is the canonical per-step audit log for the adaptive workflow. It records candidate pipeline outcomes, selected pair/order, and commit status. A step with invalid judge response keeps committed fields as NA and must not update model state. The selected endpoints i/j are the pre-orientation item indices, while A/B are the displayed / judged item indices after order assignment. Y is defined relative to displayed order: Y = 1 means A wins and Y = 0 means B wins. For cross-run audit and reuse, prefer the stable *_id columns plus unordered_key/ordered_key rather than transient integer item positions from the live state. Judge provenance, token counts, and raw_response_json are canonical step-log outputs, with raw_response_json stored as serialized character data rather than a list-column.

Core columns:

• Identity/outcome: step_id, timestamp, pair_id, i, j, i_id, j_id, A, B, A_id, B_id, unordered_key, ordered_key, Y, status.

• Judge audit: judge_backend, judge_model, judge_endpoint, judge_valid, judge_invalid_reason, llm_status_code, llm_error_message, llm_custom_id, prompt_tokens, completion_tokens, total_tokens, raw_response_json.

• Routing/scheduling: round_id, round_stage, pair_type, stage_committed_so_far, stage_quota.

• Exposure/strata: used_in_round_i, used_in_round_j, is_anchor_i, is_anchor_j, stratum_i, stratum_j, dist_stratum.

• Candidate health: is_explore_step, explore_mode, explore_reason, explore_rate_used, local_priority_mode, long_gate_pass, long_gate_reason, star_override_used, star_override_reason, candidate_starved, fallback_used, fallback_path, starvation_reason.

• Candidate counts: n_candidates_generated, n_candidates_after_hard_filters, n_candidates_after_duplicates, n_candidates_after_star_caps, n_candidates_scored.

• Endpoint diagnostics: deg_i, deg_j, recent_deg_i, recent_deg_j, mu_i, mu_j, sigma_i, sigma_j, p_ij, U0_ij.

• Star-cap diagnostics: star_cap_rejects, star_cap_reject_items.

Value

A tibble with one row per attempted step, in execution order.

See Also

adaptive_get_logs(), adaptive_round_log(), adaptive_rank_run_live()

Other adaptive logs: adaptive_get_logs(), adaptive_item_log(), adaptive_results_history(), adaptive_round_log(), summarize_items(), summarize_refits()

Examples

state <- adaptive_rank_start(c("a", "b", "c"), seed = 1)
adaptive_step_log(state)

Deterministically alternate sample order in pairs

Description

This helper takes a table of paired writing samples (with columns ID1, text1, ID2, and text2) and reverses the sample order for every second row (rows 2, 4, 6, ...). This provides a perfectly balanced reversal pattern without the randomness of randomize_pair_order().

Usage

alternate_pair_order(pairs)

Arguments

Details

This is useful when you want a fixed 50/50 mix of original and reversed pairs for bias control, benchmarking, or debugging, without relying on the random number generator or seeds.

Value

A tibble identical to pairs except that rows 2, 4, 6, ... have ID1/text1 and ID2/text2 swapped.

See Also

read_samples_df(), read_samples_dir()

Other pairing and data: make_pairs(), randomize_pair_order(), read_samples_df(), read_samples_dir(), sample_pairs(), sample_reverse_pairs()

Examples

data("example_writing_samples")
pairs <- make_pairs(example_writing_samples)

pairs_alt <- alternate_pair_order(pairs)

head(pairs[, c("ID1", "ID2")])
head(pairs_alt[, c("ID1", "ID2")])

Live Anthropic (Claude) comparison for a single pair of samples

Description

This function sends a single pairwise comparison prompt to the Anthropic Messages API (Claude models) and parses the result into a small tibble.

Usage

anthropic_compare_pair_live(
  ID1,
  text1,
  ID2,
  text2,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  tag_prefix = "<BETTER_SAMPLE>",
  tag_suffix = "</BETTER_SAMPLE>",
  api_key = NULL,
  anthropic_version = "2023-06-01",
  reasoning = c("none", "enabled"),
  include_raw = FALSE,
  include_thoughts = NULL,
  ...
)

Arguments

Details

It mirrors the behaviour and output schema of openai_compare_pair_live, but targets Anthropic's /v1/messages endpoint. The prompt template, ⁠<BETTER_SAMPLE>⁠ tag convention, and downstream parsing / BT modelling can remain unchanged.

The function is designed to work with Claude models such as Sonnet, Haiku, and Opus in the "4.5" family. You can pass any valid Anthropic model string, for example:

• "claude-sonnet-4-5"

• "claude-haiku-4-5"

• "claude-opus-4-5"

The API typically responds with a dated model string such as "claude-sonnet-4-5-20250929" in the model field.

Recommended defaults for pairwise writing comparisons

For stable, reproducible comparisons we recommend:

• reasoning = "none" with model-default sampling and max_tokens = 768 for standard pairwise scoring.

• reasoning = "enabled" when you explicitly want extended thinking; in this mode Anthropic requires temperature = 1. The default in this function is max_tokens = 2048 and thinking_budget_tokens = 1024, which satisfies the documented constraints thinking_budget_tokens >= 1024 and thinking_budget_tokens < max_tokens.

When reasoning = "enabled", this function also sends a thinking block to the Anthropic API:

"thinking": {
  "type": "enabled",
  "budget_tokens": <thinking_budget_tokens>
}

Setting include_thoughts = TRUE when reasoning = "none" is a convenient way to opt into Anthropic's extended thinking mode without changing the reasoning argument explicitly. In that case, reasoning is upgraded to "enabled", the default temperature becomes 1, and a thinking block is included in the request. When reasoning = "none" and include_thoughts is FALSE or NULL, omitted sampling parameters use the model/provider defaults.

Value

A tibble with one row and columns:

custom_id

Stable ID for the pair (pair_uid if supplied via ...; otherwise "LIVE_<ID1>_vs_<ID2>").

ID1, ID2

The sample IDs you supplied.

model

Model name reported by the API.

object_type

Anthropic object type (for example "message").

status_code

HTTP-style status code (200 if successful).

error_message

Error message if something goes wrong; otherwise NA.

thoughts

Summarised thinking / reasoning text when reasoning = "enabled" and the API returns thinking blocks; otherwise NA.

content

Concatenated text from the assistant output (excluding thinking blocks).

better_sample

"SAMPLE_1", "SAMPLE_2", or NA.

better_id

ID1 if SAMPLE_1 is chosen, ID2 if SAMPLE_2 is chosen, otherwise NA.

prompt_tokens

Prompt / input token count (if reported).

completion_tokens

Completion / output token count (if reported).

total_tokens

Total token count (reported by the API or computed as input + output tokens when not provided).

raw_response

(Optional) list-column containing the parsed JSON body.

See Also

check_llm_api_keys(), llm_compare_pair()

Other live backends: check_llm_api_keys(), gemini_compare_pair_live(), llm_compare_pair(), ollama_compare_pair_live(), openai_compare_pair_live(), submit_anthropic_pairs_live(), submit_gemini_pairs_live(), submit_llm_pairs(), submit_ollama_pairs_live(), submit_openai_pairs_live(), submit_together_pairs_live(), submit_vertex_pairs_live(), together_compare_pair_live(), vertex_compare_pair_live()

Examples

## Not run: 
# Requires ANTHROPIC_API_KEY and network access.
library(pairwiseLLM)

data("example_writing_samples", package = "pairwiseLLM")
samples <- example_writing_samples[1:2, ]

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

# Standard comparison with model-default sampling and no thinking block
res_claude <- anthropic_compare_pair_live(
  ID1               = samples$ID[1],
  text1             = samples$text[1],
  ID2               = samples$ID[2],
  text2             = samples$text[2],
  model             = "claude-sonnet-4-5",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  reasoning         = "none",
  include_raw       = FALSE
)

res_claude$better_id

# Allow more internal thinking and a longer explanation
res_claude_reason <- anthropic_compare_pair_live(
  ID1               = samples$ID[1],
  text1             = samples$text[1],
  ID2               = samples$ID[2],
  text2             = samples$text[2],
  model             = "claude-sonnet-4-5",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  reasoning         = "enabled",
  include_raw       = TRUE,
  include_thoughts  = TRUE
)

res_claude_reason$total_tokens
substr(res_claude_reason$content, 1, 200)

## End(Not run)

Create an Anthropic Message Batch

Description

This is a thin wrapper around Anthropic's /v1/messages/batches endpoint. It accepts a list of request objects (each with custom_id and params) and returns the resulting Message Batch object.

Usage

anthropic_create_batch(
  requests,
  api_key = Sys.getenv("ANTHROPIC_API_KEY"),
  anthropic_version = "2023-06-01"
)

Arguments

Details

Typically you will not call this directly; instead, use run_anthropic_batch_pipeline which builds requests from a tibble of pairs, creates the batch, polls for completion, and downloads the results.

Value

A list representing the Message Batch object returned by Anthropic. Important fields include id, processing_status, request_counts, and (after completion) results_url.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

## Not run: 
# Requires ANTHROPIC_API_KEY and network access.
library(pairwiseLLM)

data("example_writing_samples", package = "pairwiseLLM")

pairs <- example_writing_samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 2, seed = 123) |>
  randomize_pair_order(seed = 456)

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

req_tbl <- build_anthropic_batch_requests(
  pairs             = pairs,
  model             = "claude-sonnet-4-5",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl
)

requests <- lapply(seq_len(nrow(req_tbl)), function(i) {
  list(
    custom_id = req_tbl$custom_id[i],
    params    = req_tbl$params[[i]]
  )
})

batch <- anthropic_create_batch(requests = requests)
batch$id
batch$processing_status

## End(Not run)

Download Anthropic Message Batch results (.jsonl)

Description

Once a Message Batch has finished processing (status "ended"), Anthropic exposes a results_url field pointing to a .jsonl file containing one JSON object per request result.

Usage

anthropic_download_batch_results(
  batch_id,
  output_path,
  api_key = Sys.getenv("ANTHROPIC_API_KEY"),
  anthropic_version = "2023-06-01"
)

Arguments

Details

This helper downloads that file and writes it to disk. It is the Anthropic counterpart to openai_download_batch_output().

Value

Invisibly, the output_path.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

## Not run: 
# Requires ANTHROPIC_API_KEY and network access.
final <- anthropic_poll_batch_until_complete(batch$id)
jsonl_path <- tempfile(fileext = ".jsonl")
anthropic_download_batch_results(final$id, jsonl_path)

## End(Not run)

Retrieve an Anthropic Message Batch by ID

Description

This retrieves the latest state of a Message Batch using its id. It corresponds to a GET request on /v1/messages/batches/<MESSAGE_BATCH_ID>.

Usage

anthropic_get_batch(
  batch_id,
  api_key = Sys.getenv("ANTHROPIC_API_KEY"),
  anthropic_version = "2023-06-01"
)

Arguments

Value

A list representing the Message Batch object, including fields such as id, processing_status, request_counts, and (after completion) results_url.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

## Not run: 
# Requires ANTHROPIC_API_KEY and network access.
# After creating a batch:
batch <- anthropic_create_batch(requests = my_requests)
batch_id <- batch$id

latest <- anthropic_get_batch(batch_id)
latest$processing_status

## End(Not run)

Poll an Anthropic Message Batch until completion

Description

This helper repeatedly calls anthropic_get_batch until the batch's processing_status becomes "ended" or a time limit is reached. It is analogous to openai_poll_batch_until_complete() but for Anthropic's Message Batches API.

Usage

anthropic_poll_batch_until_complete(
  batch_id,
  interval_seconds = 60,
  timeout_seconds = 86400,
  api_key = Sys.getenv("ANTHROPIC_API_KEY"),
  anthropic_version = "2023-06-01",
  verbose = TRUE
)

Arguments

Value

The final Message Batch object as returned by anthropic_get_batch once processing_status == "ended" or the last object retrieved before timing out.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

## Not run: 
# Requires ANTHROPIC_API_KEY and network access.
batch <- anthropic_create_batch(requests = my_requests)
final <- anthropic_poll_batch_until_complete(batch$id, interval_seconds = 30)
final$processing_status

## End(Not run)

Build Anthropic Message Batch requests from a tibble of pairs

Description

This helper converts a tibble of writing pairs into a list of Anthropic Message Batch requests. Each request has a unique custom_id of the form "ANTH_<ID1>_vs_<ID2>" and a params object compatible with the /v1/messages API.

Usage

build_anthropic_batch_requests(
  pairs,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  reasoning = c("none", "enabled"),
  custom_id_prefix = "ANTH",
  ...
)

Arguments

Details

The function mirrors the behaviour of build_openai_batch_requests but targets Anthropic's /v1/messages/batches endpoint. It applies the same recommended defaults and reasoning constraints as anthropic_compare_pair_live:

• reasoning = "none":

  • Omitted temperature and top_p values are not sent, so the model/provider defaults apply.

  • Default max_tokens = 768, unless overridden via max_tokens in ....

• reasoning = "enabled" (extended thinking):

  • temperature must be 1. If you supply a different value in ..., this function throws an error.

  • Defaults to max_tokens = 2048 and thinking_budget_tokens = 1024, with the constraint 1024 <= thinking_budget_tokens < max_tokens. Violations of this constraint produce an error.

As a result, batches without extended thinking (reasoning = "none") use model-default sampling. When you opt into extended thinking (reasoning = "enabled"), Anthropic's requirement of temperature = 1 is enforced for all batch requests.

Value

A tibble with one row per pair and two main columns:

custom_id

Character ID of the form "<PREFIX>_<ID1>_vs_<ID2>".

params

List-column containing the Anthropic Messages API params object for each request, ready to be used in the requests array of /v1/messages/batches.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

data("example_writing_samples", package = "pairwiseLLM")

pairs <- example_writing_samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 3, seed = 123) |>
  randomize_pair_order(seed = 456)

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

# Standard batch requests without extended thinking
reqs_none <- build_anthropic_batch_requests(
  pairs             = pairs,
  model             = "claude-sonnet-4-5",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  reasoning         = "none"
)

reqs_none

# Batch requests with extended thinking
reqs_reason <- build_anthropic_batch_requests(
  pairs             = pairs,
  model             = "claude-sonnet-4-5",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  reasoning         = "enabled"
)

reqs_reason

Build Bradley-Terry comparison data from pairwise results

Description

This function converts pairwise comparison results into the three-column format commonly used for Bradley-Terry models: the first two columns contain object labels and the third column contains the comparison result (1 for a win of the first object, 0 for a win of the second).

Usage

build_bt_data(results)

Arguments

Details

It accepts either:

• legacy columns ID1, ID2, better_id, or

• canonical columns A_id, B_id, better_id.

Rows where better_id does not match either side of the pair (including NA) are excluded.

Value

A tibble with three columns:

• object1: ID from ID1

• object2: ID from ID2

• result: numeric value, 1 if better_id == ID1, 0 if better_id == ID2

Rows with invalid or missing better_id are dropped.

See Also

fit_bt_model(), summarize_bt_fit()

Other frequentist models: build_elo_data(), fit_bt_model(), fit_elo_model(), summarize_bt_fit()

Examples

results <- tibble::tibble(
  ID1       = c("S1", "S1", "S2"),
  ID2       = c("S2", "S3", "S3"),
  better_id = c("S1", "S3", "S2")
)

bt_data <- build_bt_data(results)
bt_data

# Using the example writing pairs
data("example_writing_pairs")
bt_ex <- build_bt_data(example_writing_pairs)
head(bt_ex)

Build canonical results_tbl data for Bayesian BTL MCMC

Description

Converts non-adaptive pairwise outcomes (for example, rows like example_writing_pairs with ID1, ID2, better_id) into the canonical results_tbl schema required by fit_bayes_btl_mcmc().

Usage

build_btl_results_data(
  results,
  phase = "phase2",
  backend = "non_adaptive_import",
  model = "unknown",
  iter_start = 1L,
  received_at_start = as.POSIXct("1970-01-01 00:00:00", tz = "UTC")
)

Arguments

Details

The output is deterministic and schema-valid:

• stable unordered_key / ordered_key values,

• deterministic pair_uid as "<unordered_key>#<occurrence>",

• deterministic iter and received_at sequences.

Value

A tibble in canonical results_tbl format with columns: pair_uid, unordered_key, ordered_key, A_id, B_id, better_id, winner_pos, phase, iter, received_at, backend, model.

See Also

fit_bayes_btl_mcmc()

Other Bayesian models: fit_bayes_btl_mcmc()

Examples

data("example_writing_pairs", package = "pairwiseLLM")

results_tbl <- build_btl_results_data(example_writing_pairs)
head(results_tbl)

ids <- sort(unique(c(results_tbl$A_id, results_tbl$B_id)))
ids

Build EloChoice comparison data from pairwise results

Description

This function converts pairwise comparison results into the two-column format used by the EloChoice package: one column for the winner and one for the loser of each trial.

Usage

build_elo_data(results)

Arguments

Details

It accepts either:

• legacy columns ID1, ID2, better_id, or

• canonical columns A_id, B_id, better_id.

Rows where better_id does not match either side of the pair (including NA) are excluded.

Value

A tibble with two columns:

• winner: ID of the winning sample

• loser: ID of the losing sample

Rows with invalid or missing better_id are dropped.

See Also

build_bt_data(), fit_bt_model()

Other frequentist models: build_bt_data(), fit_bt_model(), fit_elo_model(), summarize_bt_fit()

Examples

results <- tibble::tibble(
  ID1       = c("S1", "S1", "S2", "S3"),
  ID2       = c("S2", "S3", "S3", "S4"),
  better_id = c("S1", "S3", "S2", "S4")
)

elo_data <- build_elo_data(results)
elo_data

Build Gemini batch requests from a tibble of pairs

Description

This helper converts a tibble of writing pairs into a set of Gemini GenerateContent requests suitable for use with the Batch API (models/*:batchGenerateContent).

Usage

build_gemini_batch_requests(
  pairs,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  thinking_level = "low",
  custom_id_prefix = "GEM",
  temperature = NULL,
  top_p = NULL,
  top_k = NULL,
  max_output_tokens = NULL,
  service_tier = "standard",
  include_thoughts = FALSE,
  ...
)

Arguments

Details

Each pair receives a unique custom_id of the form "GEM_<ID1>_vs_<ID2>" and a corresponding request object containing the prompt and generation configuration.

Value

A tibble with one row per pair and two main columns:

custom_id

Character ID of the form "<PREFIX>_<ID1>_vs_<ID2>".

request

List-column containing the Gemini GenerateContent request object for each pair.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

data("example_writing_samples", package = "pairwiseLLM")

pairs <- example_writing_samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 3, seed = 123) |>
  randomize_pair_order(seed = 456)

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

# Dated tested Gemini Developer API configuration
reqs <- build_gemini_batch_requests(
  pairs             = pairs,
  model             = "gemini-3.5-flash-lite",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  thinking_level    = "low",
  include_thoughts  = TRUE
)

reqs

# Gemini 3 Flash example (minimal thinking)
reqs_flash <- build_gemini_batch_requests(
  pairs             = pairs,
  model             = "gemini-3-flash-preview",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  thinking_level    = "minimal",
  include_thoughts  = FALSE
)

reqs_flash

Build OpenAI batch JSONL lines for paired comparisons

Description

This helper constructs one JSON object per pair of writing samples, suitable for use with the OpenAI batch API. It supports both /v1/chat/completions and /v1/responses endpoints.

Usage

build_openai_batch_requests(
  pairs,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  endpoint = c("chat.completions", "responses"),
  temperature = NULL,
  top_p = NULL,
  logprobs = NULL,
  reasoning = NULL,
  include_thoughts = FALSE,
  request_id_prefix = "EXP"
)

Arguments

Value

A tibble with one row per pair and columns:

• custom_id: ID string used by the batch API.

• method: HTTP method ("POST").

• url: Endpoint path ("/v1/chat/completions" or "/v1/responses").

• body: List column containing the request body.

See Also

llm_submit_pairs_batch(), llm_resume_multi_batches()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

data("example_writing_samples", package = "pairwiseLLM")

pairs <- example_writing_samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 3, seed = 123) |>
  randomize_pair_order(seed = 456)

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

# 1. Basic chat.completions batch with no thoughts
batch_tbl_chat <- build_openai_batch_requests(
  pairs             = pairs,
  model             = "gpt-4.1",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  endpoint          = "chat.completions"
)

# 2. GPT-5.6 Sol Responses Batch with Reasoning
batch_tbl_resp <- build_openai_batch_requests(
  pairs = pairs,
  model = "gpt-5.6-sol",
  trait_name = td$name,
  trait_description = td$description,
  prompt_template = tmpl,
  endpoint = "responses",
  include_thoughts = TRUE, # implies reasoning="low" if not set
  reasoning = "medium"
)

batch_tbl_chat
batch_tbl_resp

Build a concrete LLM prompt from a template

Description

This function takes a prompt template (typically from set_prompt_template), a trait name and description, and two writing samples, and fills in the required placeholders.

Usage

build_prompt(template, trait_name, trait_desc, text1, text2)

Arguments

Details

The template must contain the placeholders: {TRAIT_NAME}, {TRAIT_DESCRIPTION}, {SAMPLE_1}, and {SAMPLE_2}.

Value

A single character string containing the completed prompt.

See Also

trait_description(), set_prompt_template()

Other prompts and traits: get_prompt_template(), list_prompt_templates(), register_prompt_template(), remove_prompt_template(), set_prompt_template(), trait_description()

Examples

tmpl <- set_prompt_template()
td <- trait_description("overall_quality")
prompt <- build_prompt(
  template   = tmpl,
  trait_name = td$name,
  trait_desc = td$description,
  text1      = "This is sample 1.",
  text2      = "This is sample 2."
)
cat(substr(prompt, 1, 200), "...\n")

Check configured API keys for LLM backends

Description

This function inspects the current R session for configured API keys used by pairwiseLLM. It checks for known environment variables such as OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, and VERTEX_API_KEY, and returns a small tibble summarising which keys are available.

Usage

check_llm_api_keys(verbose = TRUE)

Arguments

Details

It does not print or return the key values themselves - only whether each key is present. This makes it safe to run in logs, scripts, and shared environments.

Value

A tibble (data frame) with one row per backend and columns:

backend

Short backend identifier, e.g. "openai", "anthropic", "gemini", "vertex", "together".

service

Human-readable service name, e.g. "OpenAI", "Anthropic", "Google Gemini", "Vertex AI Gemini API", "Together.ai".

env_var

Name of the environment variable that is checked.

has_key

Logical flag indicating whether the key is set and non-empty.

See Also

llm_compare_pair(), submit_llm_pairs()

Other live backends: anthropic_compare_pair_live(), gemini_compare_pair_live(), llm_compare_pair(), ollama_compare_pair_live(), openai_compare_pair_live(), submit_anthropic_pairs_live(), submit_gemini_pairs_live(), submit_llm_pairs(), submit_ollama_pairs_live(), submit_openai_pairs_live(), submit_together_pairs_live(), submit_vertex_pairs_live(), together_compare_pair_live(), vertex_compare_pair_live()

Examples

# In an interactive session, quickly check which keys are configured:
check_llm_api_keys()

# In non-interactive scripts, you can disable messages and just use the
# result:
status <- check_llm_api_keys(verbose = FALSE)
status

Check positional preference and bootstrap reversal agreement

Description

This function diagnoses positional preference in LLM-based paired comparison data and provides a bootstrapped confidence interval for the overall agreement of forward vs. reverse comparisons.

Usage

check_positional_bias(
  consistency,
  n_boot = 1000,
  conf_level = 0.95,
  seed = NULL
)

Arguments

Details

It is designed to work with the output of compute_reverse_consistency, but will also accept a tibble that looks like its $details component.

Each row of details is one unordered pair after any duplicate judgments have been reduced to a per-direction majority by compute_reverse_consistency(). The agreement estimate and its percentile bootstrap interval therefore use unordered pairs as the unit of analysis and treat those rows as independently resampled units. This assumption may be inappropriate when pairs share items.

The direction-specific binomial tests likewise treat unordered-pair outcomes within a direction as independent. The overall test is paired: among inconsistent pairs, it compares the number for which position 1 won both presentations with the number for which position 2 won both. This is the exact conditional form of McNemar's test. It returns NA when there are no informative inconsistent pairs. A large p-value is not evidence that positional preference is absent.

Value

A list with two elements:

summary

A tibble with:

• n_pairs: number of unordered pairs

• prop_consistent: observed proportion of consistent pairs

• boot_mean: mean of bootstrap consistency proportions

• boot_lwr, boot_upr: bootstrap confidence interval

• p_sample1_main: p-value from a binomial test for the null hypothesis that SAMPLE_1 wins 50\ main (forward) comparisons

• p_sample1_rev: analogous p-value for the reverse comparisons

• p_sample1_overall: p-value from the paired exact test that position-1 and position-2 inconsistencies are equally likely

• total_pos1_wins: total number of wins by position 1 across forward + reverse comparisons

• total_comparisons: total number of valid forward + reverse comparisons included in the overall test

• n_inconsistent: number of pairs with inconsistent forward vs. reverse outcomes

• n_inconsistent_pos1_bias: among inconsistent pairs, how many times the winner is in position 1 in both directions

• n_inconsistent_pos2_bias: analogous for position 2

details

The input details tibble augmented with:

• winner_pos_main: "pos1" or "pos2" (or NA) indicating which position won in the main direction

• winner_pos_rev: analogous for the reversed direction

• is_pos1_bias: logical; TRUE if the pair is inconsistent and position 1 wins in both directions

• is_pos2_bias: analogous for position 2

See Also

compute_reverse_consistency()

Other bias and consistency: compute_reverse_consistency()

Examples

# Simple synthetic example
main <- tibble::tibble(
  ID1       = c("S1", "S1", "S2"),
  ID2       = c("S2", "S3", "S3"),
  better_id = c("S1", "S3", "S2")
)

rev <- tibble::tibble(
  ID1       = c("S2", "S3", "S3"),
  ID2       = c("S1", "S1", "S2"),
  better_id = c("S1", "S3", "S2")
)

rc <- compute_reverse_consistency(main, rev)
rc$summary

bias <- check_positional_bias(rc)
bias$summary

Compute consistency between forward and reverse pair comparisons

Description

Given two data frames of pairwise comparison results (one for the "forward" ordering of pairs, one for the "reverse" ordering), this function identifies unordered pairs that were evaluated in both directions and computes the proportion of consistent judgments.

Usage

compute_reverse_consistency(main_results, reverse_results)

Arguments

Details

Consistency is defined at the level of IDs: a pair is consistent if the same ID is selected as better in both directions. This function assumes each input contains columns ID1, ID2, and better_id, where better_id is the ID of the better sample (not "SAMPLE_1"/"SAMPLE_2").

Per-key majority agreement (duplicates supported). If a pair appears multiple times in main_results and/or reverse_results (e.g., submitted twice), this function aggregates each unordered pair key separately in each direction and takes the majority better_id. If there is a tie for the majority winner within a direction, that direction's majority winner is set to NA and the key is excluded from the consistency calculation.

The output details contains exactly one row per unordered pair key, which keeps it compatible with check_positional_bias. Columns other than ID1, ID2, and better_id are not grouping variables. Split results by model, template, trait, reasoning or thinking condition, and any intended replicate unit before calling this function; otherwise their judgments are pooled in the per-key majority.

Value

A list with two elements:

• summary: a tibble with one row and columns n_pairs, n_consistent, and prop_consistent. Here, n_pairs counts unordered pair keys with a non-missing majority winner in both directions.

• details: a tibble with one row per unordered pair key, including columns key, ID1_main, ID2_main, ID1_rev, ID2_rev, better_id_main, better_id_rev, and is_consistent. Additional columns provide vote counts and tie flags.

See Also

check_positional_bias()

Other bias and consistency: check_positional_bias()

Examples

main <- tibble::tibble(
  ID1       = c("A", "A", "X"),
  ID2       = c("B", "B", "Y"),
  better_id = c("A", "B", "X")  # duplicate A-B with disagreement
)
rev <- tibble::tibble(
  ID1       = c("B"),
  ID2       = c("A"),
  better_id = c("A")
)
compute_reverse_consistency(main, rev)$summary

Ensure only one Ollama model is loaded in memory

Description

ensure_only_ollama_model_loaded() is a small convenience helper for managing memory when working with large local models via Ollama. It inspects the current set of active models using the ⁠ollama ps⁠ command and attempts to unload any models that are not the one you specify.

Usage

ensure_only_ollama_model_loaded(model, verbose = TRUE)

Arguments

Details

This can be useful when running multiple large models (for example "mistral-small3.2:24b", "qwen3:32b", "gemma3:27b") on a single machine, where keeping all of them loaded simultaneously may exhaust GPU or system memory.

The function is intentionally conservative:

• If the ollama command is not available on the system or ⁠ollama ps⁠ returns an error or empty output, no action is taken and a message is printed when verbose = TRUE.

• If no active models are reported, no action is taken.

• Only models with names different from model are passed to ollama stop <name>.

This helper is not called automatically by the package; it is intended to be used programmatically in development scripts and ad hoc workflows before running comparisons with ollama_compare_pair_live() or submit_ollama_pairs_live().

This function relies on the ollama command-line interface being available on the system PATH. If the command cannot be executed or returns a non-zero status code, the function will issue a message (when verbose = TRUE) and return without making any changes.

The exact output format of ollama ps is treated as an implementation detail: this helper assumes that the first non-empty line is a header and that subsequent non-empty lines begin with the model name as the first whitespace-separated field. If the format changes in a future version of Ollama, parsing may fail and the function will simply fall back to doing nothing.

Because ⁠ollama stop⁠ affects the global Ollama server state for the current machine, you should only use this helper in environments where you are comfortable unloading models that might be in use by other processes.

Value

Invisibly returns a character vector containing the names of models that were requested to be unloaded (i.e., those passed to ollama stop). If no models were unloaded, an empty character vector is returned.

See Also

• ollama_compare_pair_live() for single-pair Ollama comparisons.

• submit_ollama_pairs_live() for row-wise Ollama comparisons across many pairs.

Examples

## Not run: 
# Keep only mistral-small3.2:24b loaded in Ollama, unloading any
# other active models
ensure_only_ollama_model_loaded("mistral-small3.2:24b")

## End(Not run)

Estimate LLM token usage and cost for a set of pairwise comparisons

Description

Estimate total token usage and cost for running a large set of pairwise comparisons by:

• running a small pilot on n_test pairs (live calls) to observe prompt_tokens and completion_tokens, and

• using the pilot to calibrate a prompt-bytes-to-input-token model for the remaining pairs, and

• prorating output tokens for the remaining pairs from the pilot distribution.

Usage

estimate_llm_pairs_cost(
  pairs,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  backend = c("openai", "anthropic", "gemini", "vertex", "together"),
  endpoint = c("chat.completions", "responses"),
  mode = c("live", "batch"),
  n_test = 25,
  test_strategy = c("stratified_prompt_bytes", "random", "first"),
  seed = NULL,
  cost_per_million_input,
  cost_per_million_output,
  batch_discount = 1,
  budget_quantile = 0.9,
  return_test_results = TRUE,
  return_remaining_pairs = TRUE,
  ...
)

Arguments

Details

The estimator does not require a provider tokenizer. Input tokens are estimated from the byte length of the fully constructed prompt and calibrated on the pilot's observed prompt_tokens. Expected remaining completion tokens use the arithmetic mean of usable pilot completion counts. Budget completion tokens use the type-7 sample quantile selected by budget_quantile. These are token-use estimates; the function does not supply, validate, or refresh provider prices.

Pilot pairs are selected from pairs and are excluded from remaining_pairs whether their calls succeed or fail. Calibration uses successful normalized rows with non-missing input-token counts and a missing or 200 status. Completion estimates use the analogous usable completion-token rows. With no usable input counts, calibration falls back to one token per four prompt bytes; with one row it uses a zero-intercept ratio; with multiple distinct byte lengths it uses ordinary least squares; and with multiple identical byte lengths it uses their mean token count as a constant prediction. With no usable completion counts, completion and total-cost estimates that depend on them are NA.

The observed pilot token totals are always priced as live calls. When mode = "batch", batch_discount applies only to estimated remaining input and output tokens. The returned pilot object and remaining pairs are not automatically merged into a later job result. Local Ollama models are unsupported because this function estimates token-billed API cost rather than local compute cost.

Value

An object of class "pairwiseLLM_cost_estimate", a list with:

summary

A one-row tibble with expected and budget token and cost estimates (and pilot usage).

calibration

A list describing the input-token calibration (coefficients and fit diagnostics).

test_pairs

The pilot pair subset.

pilot

Pilot results (when return_test_results = TRUE).

remaining_pairs

Remaining pairs (when return_remaining_pairs = TRUE).

See Also

make_pairs(), submit_llm_pairs(), llm_submit_pairs_batch()

Examples

## Not run: 
# Requires an API key and internet access.
data("example_writing_samples", package = "pairwiseLLM")

pairs <- example_writing_samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 50, seed = 123)

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

est <- estimate_llm_pairs_cost(
  pairs = pairs,
  backend = "openai",
  model = "gpt-4.1",
  endpoint = "chat.completions",
  trait_name = td$name,
  trait_description = td$description,
  prompt_template = tmpl,
  mode = "batch",
  batch_discount = 0.5,
  n_test = 10,
  cost_per_million_input = 0.15,
  cost_per_million_output = 0.60
)

est
est$summary

# Reuse pilot results and run only remaining pairs:
remaining <- est$remaining_pairs

## End(Not run)

Example OpenAI Batch output (JSONL lines)

Description

A small character vector containing three example lines from an OpenAI Batch API output file in JSONL format. Each element is a single JSON object representing the result for one batch request.

Usage

data("example_openai_batch_output")

Format

A character vector of length 3, where each element is a single JSON line (JSONL).

Details

The structure follows the current Batch API output schema, with fields such as id, custom_id, and a nested response object containing status_code, request_id, and a body that resembles a regular chat completion response. One line illustrates a successful comparison where <BETTER_SAMPLE>SAMPLE_1</BETTER_SAMPLE> is returned, one illustrates a case where SAMPLE_2 is preferred, and one illustrates an error case with a non-200 status.

This dataset is designed for use in examples and tests of batch output parsing functions. Typical usage is to write the lines to a temporary file and then read/parse them as a JSONL batch file.

Source

Package-author-created fictional API response fixtures. Reproduction code is in data-raw/example_writing_samples.R in the source repository.

Examples

data("example_openai_batch_output")

# Inspect the first line
cat(example_openai_batch_output[1], "\n")

# Write to a temporary .jsonl file for parsing
tmp <- tempfile(fileext = ".jsonl")
writeLines(example_openai_batch_output, con = tmp)
tmp

Example dataset of paired comparisons for writing samples

Description

A complete set of unordered paired comparison outcomes for the 20 samples in example_writing_samples. For each pair of IDs, the better_id field indicates which sample is assumed to be better, based on the quality_score in example_writing_samples.

Usage

data("example_writing_pairs")

Format

A tibble with 190 rows and 3 variables:

ID1

Character ID of the first sample in the pair.

ID2

Character ID of the second sample in the pair.

better_id

Character ID of the sample judged better in this pair (either ID1 or ID2).

Details

This dataset is useful for demonstrating functions that process paired comparisons (e.g., building Bradley-Terry data and fitting btm models) without requiring any calls to an LLM.

Source

Deterministically derived from example_writing_samples by data-raw/example_writing_samples.R in the source repository.

Examples

data("example_writing_pairs")
head(example_writing_pairs)

Example canonical results table for writing comparisons

Description

Canonical results_tbl representation of example_writing_pairs, intended for direct use with fit_bayes_btl_mcmc and other functions that require adaptive schema-compatible results input.

Usage

data("example_writing_results")

Format

A tibble with 190 rows and 12 variables:

pair_uid

Deterministic pair attempt ID.

unordered_key

Unordered pair key ("min:max").

ordered_key

Ordered pair key ("A_id:B_id").

A_id

Character ID in first position.

B_id

Character ID in second position.

better_id

Character ID judged better in this comparison.

winner_pos

Integer winner position (1L or 2L).

phase

Phase label.

iter

Integer step index.

received_at

POSIXct timestamp in UTC.

backend

Backend label for provenance.

model

Model label for provenance.

Source

Deterministically derived from example_writing_pairs by data-raw/example_writing_samples.R in the source repository.

Examples

data("example_writing_results")
head(example_writing_results)

Example dataset of writing samples

Description

A small set of 20 writing samples on the topic "Why is writing assessment difficult?", intended for use in examples and tests involving pairing and LLM-based comparisons. The samples vary in quality, approximately from very weak to very strong, and a simple numeric quality score is included to support simulated comparison outcomes.

Usage

data("example_writing_samples")

Format

A tibble with 20 rows and 3 variables:

ID

Character ID for each sample (e.g., "S01").

text

Character string with the writing sample.

quality_score

Integer from 1 to 10 indicating the intended relative quality of the sample (higher = better).

Source

Package-author-created synthetic examples. Reproduction code is in data-raw/example_writing_samples.R in the source repository.

Examples

data("example_writing_samples")
example_writing_samples

Synthetic Writing Samples with Controlled Quality Levels (N = 1000)

Description

A synthetic dataset of 1,000 short writing samples generated by a large language model for use in pairwise comparison and ranking experiments.

Usage

data("example_writing_samples1000")

Format

A tibble with 1,000 rows and 7 variables:

ID

Character. Unique sample identifier (S0001–S1000).

text

Character. The writing sample (approximately 120–180 words).

quality_level

Integer. Intended quality level used during generation (1–20).

theta_true

Numeric. Centered latent-quality proxy derived from quality_level.

prompt_id

Character. Identifier for the generation prompt template.

model

Character. Language model used to generate the samples.

created_at

POSIXct. Timestamp (UTC) when the samples were generated.

Details

Samples are generated in 20 discrete quality levels (1 = lowest, 20 = highest), with multiple responses per level. Quality levels are intended to represent overlapping ranges of overall writing quality rather than a strict total ordering, allowing for realistic noise and near-ties in pairwise judgments.

All samples respond to the same writing prompt to avoid topic effects. The dataset is primarily intended for benchmarking ranking models and for comparing random versus adaptive pair selection strategies under limited judgment budgets.

The column theta_true provides a centered numeric proxy for the latent quality dimension derived from quality_level. This proxy is intended for evaluation purposes (e.g., rank recovery or correlation) and does not imply a perfectly ordered ground truth at the individual-sample level.

Source

Generated via live OpenAI API calls using a controlled, bucketed quality prompt. See data-raw/generate_example_writing_samples1000.R for details.

Examples

data(example_writing_samples1000)
head(example_writing_samples1000)

Full Bayesian BTL inference via CmdStanR (adaptive-compatible)

Description

Runs full Bayesian posterior inference for a Bradley–Terry–Luce (BTL) style model using the package’s CmdStan machinery, but in a standalone (non-adaptive) context. The function is designed so downstream diagnostics and reporting can reuse the existing adaptive summary tools (notably summarize_items() and summarize_refits()) without requiring new summary functions.

Usage

fit_bayes_btl_mcmc(
  results,
  ids,
  model_variant = "btl_e_b",
  cmdstan = list(iter_warmup = 1000, iter_sampling = 1000, seed = NULL, core_fraction =
    0.8),
  pair_counts = NULL,
  subset_method = c("first", "sample"),
  seed = NULL,
  inference_contract = NULL
)

Arguments

Details

Internally, the function can optionally refit the model on increasing subsets of the observed comparisons (via pair_counts). Each refit is treated as a "refit" in the adaptive logging sense, producing:

• one round-log row per refit (compatible with round_log_schema()),

• one item-log table per refit (compatible with .adaptive_item_log_schema()).

Value

A list with:

item_log_list

List of item-log tables, one per refit, matching the canonical adaptive item log schema. This is the preferred structure for reuse with summarize_items().

item_summary

A single tibble formed by row-binding item_log_list (kept for backward compatibility). Each row corresponds to an item within a refit; refit_id identifies the refit.

round_log

Tibble matching the canonical adaptive round log schema (one row per refit).

fits

List of BTL fit contracts (one per refit).

fit

Single fit contract (only when one refit is run).

See Also

build_btl_results_data()

Other Bayesian models: build_btl_results_data()

Examples

## Not run: 
results <- tibble::tibble(
  pair_uid = "A:B#1",
  unordered_key = "A:B",
  ordered_key = "A:B",
  A_id = "A",
  B_id = "B",
  better_id = "A",
  winner_pos = 1L,
  phase = "phase2",
  iter = 1L,
  received_at = as.POSIXct("2026-01-01 00:00:00", tz = "UTC"),
  backend = "openai",
  model = "gpt-test"
)

fit <- fit_bayes_btl_mcmc(
  results,
  ids = c("A", "B"),
  model_variant = "btl_e_b"
)

# Generate summaries
summarize_refits(fit)
summarize_items(fit)

## End(Not run)

Fit a Bradley–Terry model with sirt and fallback to BradleyTerry2

Description

This function fits a Bradley–Terry paired-comparison model to data prepared by build_bt_data. It supports two modeling engines:

• sirt: btm — the preferred engine, which produces ability estimates, standard errors, and MLE reliability.

• BradleyTerry2: BTm — used as a fallback if sirt is unavailable or fails; computes ability estimates and standard errors, but not reliability.

Usage

fit_bt_model(
  bt_data,
  engine = c("auto", "sirt", "BradleyTerry2"),
  verbose = TRUE,
  ...
)

Arguments

Details

When engine = "auto" (the default), the function attempts sirt first and automatically falls back to BradleyTerry2 only if necessary. In all cases, the output format is standardized, so downstream code can rely on consistent fields.

The input bt_data must contain exactly three columns:

1. object1: character ID for the first item in the pair

2. object2: character ID for the second item

3. result: numeric indicator (1 = object1 wins, 0 = object2 wins)

Ability estimates (theta) represent latent "writing quality" parameters on a log-odds scale. Standard errors are included for both modeling engines. MLE reliability is only available from sirt.

Value

A list with the following elements:

engine

The engine actually used ("sirt" or "BradleyTerry2").

fit

The fitted model object.

theta

A tibble with columns:

• ID: object identifier

• theta: estimated ability parameter

• se: standard error of theta

reliability

MLE reliability (sirt engine only). NA for BradleyTerry2 models.

See Also

build_bt_data(), summarize_bt_fit()

Other frequentist models: build_bt_data(), build_elo_data(), fit_elo_model(), summarize_bt_fit()

Examples

# Example using built-in comparison data
data("example_writing_pairs")
bt <- build_bt_data(example_writing_pairs)

if (requireNamespace("sirt", quietly = TRUE)) {
  fit1 <- fit_bt_model(bt, engine = "sirt")
}
if (requireNamespace("BradleyTerry2", quietly = TRUE)) {
  fit2 <- fit_bt_model(bt, engine = "BradleyTerry2")
}

Fit an EloChoice model to pairwise comparison data

Description

This function fits an Elo-based paired-comparison model using the EloChoice package. It is intended to complement fit_bt_model by providing an alternative scoring framework based on Elo ratings rather than Bradley–Terry models.

Usage

fit_elo_model(elo_data, runs = 5, verbose = FALSE, ...)

Arguments

Details

The input elo_data must contain two columns:

1. winner: ID of the winning sample in each pairwise trial

2. loser: ID of the losing sample in each trial

These can be created from standard pairwise comparison output using build_elo_data.

Internally, this function calls:

• elochoice — to estimate Elo ratings using repeated randomization of trial order;

• reliability — to compute unweighted and weighted reliability indices as described in Clark et al. (2018).

If the EloChoice package is not installed, a helpful error message is shown telling the user how to install it.

The returned object mirrors the structure of fit_bt_model for consistency across scoring engines:

• engine — always "EloChoice".

• fit — the raw "elochoice" object returned by EloChoice::elochoice().

• elo — a tibble with columns:

  • ID: sample identifier

  • elo: estimated Elo rating

  (Unlike Bradley–Terry models, EloChoice does not provide standard errors for these ratings, so none are returned.)

• reliability — the mean unweighted reliability index (mean proportion of “upsets” across randomizations).

• reliability_weighted — the mean weighted reliability index (weighted version of the upset measure).

Value

A named list with components:

engine

Character scalar identifying the scoring engine ("EloChoice").

fit

The "elochoice" model object.

elo

A tibble with columns ID and elo.

reliability

Numeric scalar: mean unweighted reliability index.

reliability_weighted

Numeric scalar: mean weighted reliability index.

References

Clark AP, Howard KL, Woods AT, Penton-Voak IS, Neumann C (2018). "Why rate when you could compare? Using the 'EloChoice' package to assess pairwise comparisons of perceived physical strength." PLOS ONE, 13(1), e0190393. doi:10.1371/journal.pone.0190393.

See Also

build_bt_data(), fit_bt_model()

Other frequentist models: build_bt_data(), build_elo_data(), fit_bt_model(), summarize_bt_fit()

Examples

data("example_writing_pairs", package = "pairwiseLLM")

elo_data <- build_elo_data(example_writing_pairs)

if (requireNamespace("EloChoice", quietly = TRUE)) {
  fit <- fit_elo_model(elo_data, runs = 5, verbose = FALSE)
  fit$elo
  fit$reliability
  fit$reliability_weighted
}

Live Google Gemini comparison for a single pair of samples

Description

This function sends a single pairwise comparison prompt to the Google Gemini Generative Language API (Gemini 3 Pro / Flash) and parses the result into a one-row tibble that mirrors the structure used for OpenAI / Anthropic live calls.

Usage

gemini_compare_pair_live(
  ID1,
  text1,
  ID2,
  text2,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  api_key = NULL,
  thinking_level = "low",
  temperature = NULL,
  top_p = NULL,
  top_k = NULL,
  max_output_tokens = NULL,
  service_tier = "standard",
  api_version = "v1beta",
  include_raw = FALSE,
  include_thoughts = FALSE,
  pair_uid = NULL,
  ...
)

Arguments

Details

It expects the prompt template to instruct the model to choose exactly one of SAMPLE_1 or SAMPLE_2 and wrap the decision in <BETTER_SAMPLE> tags, for example:

<BETTER_SAMPLE>SAMPLE_1</BETTER_SAMPLE>

or

<BETTER_SAMPLE>SAMPLE_2</BETTER_SAMPLE>

If include_thoughts = TRUE, the function additionally requests Gemini's explicit chain-of-thought style reasoning ("thoughts") via the thinkingConfig block and stores it in a separate thoughts column, while still using the final answer content to detect the ⁠<BETTER_SAMPLE>⁠ tag.

Value

A tibble with one row and columns:

• custom_id - stable ID for the pair (pair_uid if supplied).

• ID1, ID2 - provided sample IDs.

• model - model name returned by the API (or the requested model).

• object_type - "generateContent" on success, otherwise NA.

• status_code - HTTP status code (200 on success).

• error_message - error message for failures, otherwise NA.

• thoughts - explicit chain-of-thought style reasoning text if include_thoughts = TRUE and the model returns it; otherwise NA.

• content - concatenated text of the assistant's final answer (used to locate the ⁠<BETTER_SAMPLE>⁠ tag).

• better_sample - "SAMPLE_1", "SAMPLE_2", or NA.

• better_id - ID1 if SAMPLE_1 is chosen, ID2 if SAMPLE_2, or NA.

• prompt_tokens, completion_tokens, total_tokens - usage counts if reported by the API, otherwise NA_real_.

See Also

check_llm_api_keys(), llm_compare_pair()

Other live backends: anthropic_compare_pair_live(), check_llm_api_keys(), llm_compare_pair(), ollama_compare_pair_live(), openai_compare_pair_live(), submit_anthropic_pairs_live(), submit_gemini_pairs_live(), submit_llm_pairs(), submit_ollama_pairs_live(), submit_openai_pairs_live(), submit_together_pairs_live(), submit_vertex_pairs_live(), together_compare_pair_live(), vertex_compare_pair_live()

Examples

# Requires:
# - GEMINI_API_KEY set in your environment
# - Internet access
# - Billable Gemini API usage
## Not run: 
td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

# Dated tested Gemini Developer API configuration
res <- gemini_compare_pair_live(
  ID1               = "S01",
  text1             = "Text 1",
  ID2               = "S02",
  text2             = "Text 2",
  model             = "gemini-3.5-flash-lite",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  thinking_level    = "low",
  include_thoughts  = FALSE,
  include_raw       = FALSE
)

res
res$better_id

# Gemini 3 Flash example (minimal thinking)
res_flash <- gemini_compare_pair_live(
  ID1               = "S01",
  text1             = "Text 1",
  ID2               = "S02",
  text2             = "Text 2",
  model             = "gemini-3-flash-preview",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  thinking_level    = "minimal",
  include_thoughts  = FALSE,
  include_raw       = FALSE
)

res_flash

## End(Not run)

Create a Gemini Batch job from request objects

Description

This is a thin wrapper around the REST endpoint /v1beta/models/<MODEL>:batchGenerateContent. It accepts a list of GenerateContent request objects and returns the created Batch job.

Usage

gemini_create_batch(
  requests,
  model,
  api_key = Sys.getenv("GEMINI_API_KEY"),
  api_version = "v1beta",
  display_name = NULL
)

Arguments

Details

Typically you will not call this directly; instead, use run_gemini_batch_pipeline which builds requests from a tibble of pairs, creates the batch, polls for completion, and parses the results.

Value

A list representing the Batch job object returned by Gemini. Important fields include name, metadata$state, and (after completion) response$inlinedResponses or response$responsesFile.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

# --- Offline preparation: build GenerateContent requests ---

data("example_writing_samples", package = "pairwiseLLM")

pairs <- example_writing_samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 2, seed = 123)

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

batch_tbl <- build_gemini_batch_requests(
  pairs             = pairs,
  model             = "gemini-3.5-flash-lite",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  thinking_level    = "low"
)

# Extract the list of request objects
requests <- batch_tbl$request

# Inspect a single GenerateContent request (purely local)
requests[[1]]

# --- Online step: create the Gemini Batch job ---
# Requires network access and a valid Gemini API key.
## Not run: 
batch <- gemini_create_batch(
  requests = requests,
  model    = "gemini-3.5-flash-lite"
)

batch$name
batch$metadata$state

## End(Not run)

Download Gemini Batch results to a JSONL file

Description

For inline batch requests, Gemini returns results under response$inlinedResponses$inlinedResponses. In the v1beta REST API this often comes back as a data frame with one row per request and a "response" column, where each "response" is itself a data frame of GenerateContentResponse objects.

Usage

gemini_download_batch_results(
  batch,
  requests_tbl,
  output_path,
  api_key = Sys.getenv("GEMINI_API_KEY"),
  api_version = "v1beta"
)

Arguments

Details

This helper writes those results to a local .jsonl file where each line is a JSON object of the form:

{"custom_id": "<GEM_ID1_vs_ID2>",
 "result": {
   "type": "succeeded",
   "response": { ... GenerateContentResponse ... }
 }}

or, when an error occurred:

{"custom_id": "<GEM_ID1_vs_ID2>",
 "result": {
   "type": "errored",
   "error": { ... }
 }}

Value

Invisibly returns output_path.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

# This example requires a Gemini API key and network access.
# It assumes you have already created and run a Gemini batch job.
## Not run: 
# Name of an existing Gemini batch
batch_name <- "batches/123456"

# Requests table used to create the batch (must include custom_id)
requests_tbl <- tibble::tibble(
  custom_id = c("GEM_S01_vs_S02", "GEM_S03_vs_S04")
)

# Download inline batch results to a local JSONL file
out_file <- tempfile(fileext = ".jsonl")

gemini_download_batch_results(
  batch        = batch_name,
  requests_tbl = requests_tbl,
  output_path  = out_file
)

# Inspect the downloaded JSONL
readLines(out_file, warn = FALSE)

## End(Not run)

Retrieve a Gemini Batch job by name

Description

This retrieves the latest state of a Batch job using its name as returned by gemini_create_batch.

Usage

gemini_get_batch(
  batch_name,
  api_key = Sys.getenv("GEMINI_API_KEY"),
  api_version = "v1beta"
)

Arguments

Details

It corresponds to a GET request on /v1beta/<BATCH_NAME>, where BATCH_NAME is a string such as "batches/123456".

Value

A list representing the Batch job object.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

# Offline: basic batch name validation / object you would pass
batch_name <- "batches/123456"

# Online: retrieve the batch state from Gemini (requires API key + network)
## Not run: 
batch <- gemini_get_batch(batch_name = batch_name)
batch$name
batch$metadata$state

## End(Not run)

Poll a Gemini Batch job until completion

Description

This helper repeatedly calls gemini_get_batch until the batch's metadata$state enters a terminal state or a time limit is reached. For the REST API, states have the form "BATCH_STATE_*".

Usage

gemini_poll_batch_until_complete(
  batch_name,
  interval_seconds = 60,
  timeout_seconds = 86400,
  api_key = Sys.getenv("GEMINI_API_KEY"),
  api_version = "v1beta",
  verbose = TRUE
)

Arguments

Value

The final Batch job object as returned by gemini_get_batch.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

# Offline: polling parameters and batch name are plain R objects
batch_name <- "batches/123456"

# Online: poll until the batch reaches a terminal state (requires network)
## Not run: 
final_batch <- gemini_poll_batch_until_complete(
  batch_name       = batch_name,
  interval_seconds = 10,
  timeout_seconds  = 600,
  verbose          = TRUE
)
final_batch$metadata$state

## End(Not run)

Retrieve a named prompt template

Description

This function retrieves a prompt template from either:

• the user registry (see register_prompt_template), or

• a built-in template stored under inst/templates.

Usage

get_prompt_template(name = "default")

Arguments

Details

The function first checks user-registered templates, then looks for a built-in text file inst/templates/<name>.txt. The special name "default" falls back to set_prompt_template() when no user-registered or built-in template is found.

Value

A single character string containing the prompt template.

See Also

register_prompt_template, list_prompt_templates, remove_prompt_template

Other prompts and traits: build_prompt(), list_prompt_templates(), register_prompt_template(), remove_prompt_template(), set_prompt_template(), trait_description()

Examples

# Get the built-in default template
tmpl_default <- get_prompt_template("default")

# List available template names
list_prompt_templates()

List available prompt templates

Description

This function lists template names that are available either as built-in text files under inst/templates or as user-registered templates in the current R session.

Usage

list_prompt_templates(include_builtin = TRUE, include_registered = TRUE)

Arguments

Details

Built-in templates are identified by files named <name>.txt within inst/templates. For example, a file inst/templates/minimal.txt will be listed as "minimal".

Value

A sorted character vector of unique template names.

See Also

trait_description(), set_prompt_template()

Other prompts and traits: build_prompt(), get_prompt_template(), register_prompt_template(), remove_prompt_template(), set_prompt_template(), trait_description()

Examples

list_prompt_templates()

Backend-agnostic live comparison for a single pair of samples

Description

llm_compare_pair() is a thin wrapper around backend-specific comparison functions. It currently supports the "openai", "anthropic", "gemini", "vertex", "together", and "ollama" backends and forwards the call to the appropriate live comparison helper:

• "openai" → openai_compare_pair_live()

• "anthropic" → anthropic_compare_pair_live()

• "gemini" → gemini_compare_pair_live()

• "vertex" → vertex_compare_pair_live()

• "together" → together_compare_pair_live()

• "ollama" → ollama_compare_pair_live()

Usage

llm_compare_pair(
  ID1,
  text1,
  ID2,
  text2,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  backend = c("openai", "anthropic", "gemini", "vertex", "together", "ollama"),
  endpoint = c("chat.completions", "responses"),
  api_key = NULL,
  include_raw = FALSE,
  ...
)

Arguments

Details

All backends are expected to return a tibble with a compatible structure, including:

• custom_id, ID1, ID2

• model, object_type, status_code, error_message

• thoughts (reasoning / thinking text when available)

• content (visible assistant output)

• better_sample, better_id

• prompt_tokens, completion_tokens, total_tokens

For the "openai" backend, the endpoint argument controls whether the Chat Completions API ("chat.completions") or the Responses API ("responses") is used. For the "anthropic", "gemini", "vertex", "together", and "ollama" backends, endpoint is currently ignored and the default live API for that provider is used.

Value

A tibble with one row and the same columns as the underlying backend-specific live helper (for example openai_compare_pair_live() for "openai"). All backends are intended to return a compatible structure including thoughts, content, and token counts.

See Also

• openai_compare_pair_live(), anthropic_compare_pair_live(), gemini_compare_pair_live(), vertex_compare_pair_live(), together_compare_pair_live(), and ollama_compare_pair_live() for backend-specific implementations.

• submit_llm_pairs() for row-wise comparisons over a tibble of pairs.

• build_bt_data() and fit_bt_model() for Bradley–Terry modelling of comparison results.

Other live backends: anthropic_compare_pair_live(), check_llm_api_keys(), gemini_compare_pair_live(), ollama_compare_pair_live(), openai_compare_pair_live(), submit_anthropic_pairs_live(), submit_gemini_pairs_live(), submit_llm_pairs(), submit_ollama_pairs_live(), submit_openai_pairs_live(), submit_together_pairs_live(), submit_vertex_pairs_live(), together_compare_pair_live(), vertex_compare_pair_live()

Examples

## Not run: 
# Requires an API key for the chosen cloud backend. For OpenAI, set
# OPENAI_API_KEY in your environment. Running these examples will incur
# API usage costs.
#
# For local Ollama use, an Ollama server must be running and the models
# must be pulled in advance. No API key is required for the `"ollama"`
# backend.

data("example_writing_samples", package = "pairwiseLLM")
samples <- example_writing_samples[1:2, ]

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

# Single live comparison using the OpenAI backend and chat.completions
res_live <- llm_compare_pair(
  ID1               = samples$ID[1],
  text1             = samples$text[1],
  ID2               = samples$ID[2],
  text2             = samples$text[2],
  model             = "gpt-4.1",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  backend           = "openai",
  endpoint          = "chat.completions",
  temperature       = 0
)

res_live$better_id

# Using the OpenAI responses endpoint with gpt-5.1 and reasoning = "low"
res_live_gpt5 <- llm_compare_pair(
  ID1               = samples$ID[1],
  text1             = samples$text[1],
  ID2               = samples$ID[2],
  text2             = samples$text[2],
  model             = "gpt-5.1",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  backend           = "openai",
  endpoint          = "responses",
  reasoning         = "low",
  include_thoughts  = TRUE,
  temperature       = NULL,
  top_p             = NULL,
  logprobs          = NULL,
  include_raw       = TRUE
)

str(res_live_gpt5$raw_response[[1]], max.level = 2)

# Example: single live comparison using a local Ollama backend
res_ollama <- llm_compare_pair(
  ID1 = samples$ID[1],
  text1 = samples$text[1],
  ID2 = samples$ID[2],
  text2 = samples$text[2],
  model = "mistral-small3.2:24b",
  trait_name = td$name,
  trait_description = td$description,
  prompt_template = tmpl,
  backend = "ollama",
  host = getOption(
    "pairwiseLLM.ollama_host",
    "http://127.0.0.1:11434"
  ),
  think = FALSE
)

res_ollama$better_id

## End(Not run)

Extract results from a pairwiseLLM batch object

Description

Helper to extract the parsed results tibble from a batch object returned by llm_submit_pairs_batch(). This is a thin wrapper around the results element returned by backend-specific batch pipelines and is designed to be forward-compatible with future, more asynchronous batch workflows.

Usage

llm_download_batch_results(x, ...)

Arguments

Value

A tibble containing batch comparison results in the standard pairwiseLLM schema.

See Also

llm_submit_pairs_batch(), llm_submit_pairs_multi_batch()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

## Not run: 
# Requires running a provider batch job first (API key + internet + cost).

batch <- llm_submit_pairs_batch(
  pairs             = tibble::tibble(
    ID1   = "S01",
    text1 = "Text 1",
    ID2   = "S02",
    text2 = "Text 2"
  ),
  backend           = "openai",
  model             = "gpt-4.1",
  trait_name        = trait_description("overall_quality")$name,
  trait_description = trait_description("overall_quality")$description,
  prompt_template   = set_prompt_template()
)

res <- llm_download_batch_results(batch)
res

## End(Not run)

Resume polling and download results for multiple batch jobs

Description

This function takes the output of llm_submit_pairs_multi_batch() (or a previously written registry CSV) and polls each batch until completion, downloading and parsing results as they finish. It implements a conservative polling loop with a configurable interval between rounds and a small delay between individual jobs to reduce the risk of API rate‑limit errors. The httr2 retry wrapper is still invoked for each API call, so transient HTTP errors will be retried with exponential back‑off.

Usage

llm_resume_multi_batches(
  jobs = NULL,
  output_dir = NULL,
  interval_seconds = 60,
  per_job_delay = 2,
  write_results_csv = FALSE,
  keep_jsonl = TRUE,
  write_registry = FALSE,
  tag_prefix = "<BETTER_SAMPLE>",
  tag_suffix = "</BETTER_SAMPLE>",
  verbose = FALSE,
  write_combined_csv = FALSE,
  combined_csv_path = NULL,
  openai_max_retries = 3
)

Arguments

Value

A list with four elements: jobs, the updated jobs list with each element containing parsed results and a done flag; combined, a tibble obtained by binding all completed results (NULL if no batches completed); failed_attempts, a tibble of failed attempts captured during normalization; and batch_failures, a tibble describing batches that reached a terminal non-success status. If write_results_csv is TRUE, the combined tibble is still returned in memory. If write_combined_csv is TRUE, the combined tibble is also written to a CSV file on disk (see combined_csv_path for details) but is still returned in memory.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

# Continuing the example from llm_submit_pairs_multi_batch():
# After submitting multiple batches, resume polling and combine the results.
## Not run: 
# Suppose `outdir` is the directory where batch files were written and
# `jobs` is the list of job metadata returned by llm_submit_pairs_multi_batch().

results <- llm_resume_multi_batches(
  jobs               = jobs,
  output_dir         = outdir,
  interval_seconds   = 60,
  per_job_delay      = 2,
  write_results_csv  = TRUE,
  keep_jsonl         = FALSE,
  write_registry     = TRUE,
  verbose            = TRUE,
  write_combined_csv = TRUE
)

# The combined results are available in the `combined` element
print(results$combined)

## End(Not run)

Submit pairs to an LLM backend via batch API

Description

llm_submit_pairs_batch() is a backend-agnostic front-end for running provider batch pipelines (OpenAI, Anthropic, Gemini). Vertex, Together.ai, and Ollama are supported only for live comparisons in this series.

It mirrors submit_llm_pairs() but uses the provider batch APIs under the hood via run_openai_batch_pipeline(), run_anthropic_batch_pipeline(), and run_gemini_batch_pipeline().

For OpenAI, this helper will by default:

• Use the chat.completions batch style for most models, and

• Automatically switch to the responses style endpoint when:

  • model is in the GPT-5 series (including gpt-5, gpt-5-mini, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, and date-stamped GPT-5.x variants), and

  • either include_thoughts = TRUE or a reasoning effort is supplied in ... (for GPT-5, reasoning = "none" maps to "minimal").

Sampling defaults: For OpenAI, omitted temperature and top_p values are not added to the request, so the model/provider defaults apply. Reasoning modes that do not support sampling parameters continue to require them to be NULL.

For Anthropic, standard and date-stamped model names (e.g. "claude-sonnet-4-5-20250929") are supported. This helper delegates temperature and extended-thinking behaviour to run_anthropic_batch_pipeline() and build_anthropic_batch_requests(), which apply the following rules:

• When reasoning = "none" (no extended thinking), omitted temperature and top_p values use the model/provider defaults.

• When reasoning = "enabled" (extended thinking), Anthropic requires temperature = 1. If you supply a different value in ..., an error is raised. Default values in this mode are max_tokens = 2048 and thinking_budget_tokens = 1024, subject to ⁠1024 <= thinking_budget_tokens < max_tokens⁠.

• Setting include_thoughts = TRUE while leaving reasoning = "none" causes run_anthropic_batch_pipeline() to upgrade to reasoning = "enabled", which implies temperature = 1 for the batch.

For Gemini, this helper simply forwards include_thoughts and other arguments to run_gemini_batch_pipeline(), which is responsible for interpreting any thinking-related options.

Currently, this function synchronously runs the full batch pipeline for each backend (build requests, create batch, poll until complete, download results, parse). The returned object contains both metadata and a normalized results tibble. See llm_download_batch_results() to extract the results.

Usage

llm_submit_pairs_batch(
  pairs,
  backend = c("openai", "anthropic", "gemini"),
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  include_thoughts = FALSE,
  include_raw = FALSE,
  ...
)

Arguments

Value

A list of class "pairwiseLLM_batch" containing at least:

• backend: the backend identifier ("openai", "anthropic", "gemini"),

• batch_input_path: path to the JSONL request file (if applicable),

• batch_output_path: path to the JSONL output file (if applicable),

• batch: provider-specific batch object (e.g., job metadata),

• results: a tibble of parsed comparison results in the standard pairwiseLLM schema.

• failed_attempts: a tibble of failed attempts captured during normalization (empty when no failures are observed).

Additional fields returned by the backend-specific pipeline functions are preserved.

See Also

llm_download_batch_results(), llm_submit_pairs_multi_batch()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

# Requires:
# - Internet access
# - Provider API key set in your environment (OPENAI_API_KEY /
#   ANTHROPIC_API_KEY / GEMINI_API_KEY)
# - Billable API usage
## Not run: 
pairs <- tibble::tibble(
  ID1   = c("S01", "S03"),
  text1 = c("Text 1", "Text 3"),
  ID2   = c("S02", "S04"),
  text2 = c("Text 2", "Text 4")
)

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

# OpenAI batch
batch_openai <- llm_submit_pairs_batch(
  pairs             = pairs,
  backend           = "openai",
  model             = "gpt-5-mini",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  include_thoughts  = FALSE,
  service_tier      = "flex"
)
res_openai <- llm_download_batch_results(batch_openai)

# Anthropic batch
batch_anthropic <- llm_submit_pairs_batch(
  pairs             = pairs,
  backend           = "anthropic",
  model             = "claude-sonnet-4-5",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  include_thoughts  = FALSE
)
res_anthropic <- llm_download_batch_results(batch_anthropic)

# Gemini batch
batch_gemini <- llm_submit_pairs_batch(
  pairs             = pairs,
  backend           = "gemini",
  model             = "gemini-3.5-flash-lite",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  include_thoughts  = TRUE
)
res_gemini <- llm_download_batch_results(batch_gemini)

## End(Not run)

Multi‑batch submission and polling wrappers

Description

These functions provide higher‑level wrappers around the existing provider‑specific batch APIs in pairwiseLLM. They allow a large tibble of pairwise comparisons to be automatically split into multiple batch jobs, submitted concurrently (without polling), recorded in a registry for safe resumption, and later polled until completion and merged into a single results data frame. They do not modify any of the underlying API functions such as run_openai_batch_pipeline() or run_anthropic_batch_pipeline(), but orchestrate these calls to support resilient multi‑batch workflows.

Usage

llm_submit_pairs_multi_batch(
  pairs,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  backend = c("openai", "anthropic", "gemini"),
  batch_size = NULL,
  n_segments = NULL,
  output_dir = tempfile("llm_multi_batch_"),
  write_registry = FALSE,
  keep_jsonl = TRUE,
  verbose = FALSE,
  ...,
  openai_max_retries = 3
)

Arguments

Value

A list with two elements: jobs, a list of per‑batch metadata (similar to the example in the advanced vignette), and registry, a tibble summarising all jobs. The registry contains columns segment_index, provider, model, batch_id, batch_input_path, batch_output_path, csv_path, pairs_path, done, and results (initialized to NULL). If write_registry is TRUE, the tibble is also written to disk as jobs_registry.csv.

llm_submit_pairs_multi_batch()

Splits a tibble of comparison pairs into chunks and submits one batch per chunk using the appropriate provider pipeline. Each batch is created with poll = FALSE, so the function returns immediately after the batch jobs have been created. Metadata for each batch—including the batch_id, provider type, and input/output file paths—is collected and (optionally) written to a CSV registry for later resumption.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), openai_create_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

# Example: split a small set of pairs into five segments, submit
# them to the Gemini backend, and then poll and combine the results.
# Requires a funded API key and internet access.
## Not run: 
# Construct ten random pairs from the example writing samples
set.seed(123)
pairs <- sample_pairs(example_writing_samples, n_pairs = 10)

# Directory to store batch files and results
outdir <- tempfile("multi_batch_example_")

# Submit the pairs in five batches.  We write the registry to disk
# and print progress messages as each batch is created.
job_info <- llm_submit_pairs_multi_batch(
  pairs             = pairs,
  model             = "gemini-3.5-flash-lite",
  trait_name        = "writing_quality",
  trait_description = "Which text shows better writing quality?",
  n_segments        = 5,
  output_dir        = outdir,
  write_registry    = TRUE,
  verbose           = TRUE
)

# Resume polling until all batches complete.  The per-batch and
# combined results are written to CSV files, the registry is
# refreshed on disk, and progress messages are printed.
results <- llm_resume_multi_batches(
  jobs               = job_info$jobs,
  output_dir         = outdir,
  interval_seconds   = 60,
  per_job_delay      = 2,
  write_results_csv  = TRUE,
  keep_jsonl         = FALSE,
  write_registry     = TRUE,
  verbose            = TRUE,
  write_combined_csv = TRUE
)

# Access the combined results tibble
head(results$combined)

## End(Not run)

Load an adaptive session from disk.

Description

Load an adaptive session from disk.

Usage

load_adaptive_session(session_dir)

Arguments

Details

Restores a persisted Adaptive state and revalidates basic invariants such as schema version, required state fields, and index ranges in step_log. If per-refit item logs are found on disk, they are loaded into state$item_log and persistence is marked as enabled. Resume uses strict schema validation for canonical logs; incompatible saved schemas abort with explicit errors.

Value

An adaptive_state object ready for resume.

See Also

save_adaptive_session(), validate_session_dir(), adaptive_rank_resume()

Other adaptive persistence: save_adaptive_session(), validate_session_dir()

Examples

dir <- tempfile("pwllm-session-")
state <- adaptive_rank_start(c("a", "b", "c"), seed = 1)
save_adaptive_session(state, dir, overwrite = TRUE)
restored <- load_adaptive_session(dir)
summarize_adaptive(restored)

Build an LLM judge function for adaptive ranking

Description

Creates a judge function compatible with adaptive_rank_run_live() by wrapping llm_compare_pair() and converting provider responses into adaptive binary outcomes (Y in ⁠{0,1}⁠).

Usage

make_adaptive_judge_llm(
  backend = c("openai", "anthropic", "gemini", "vertex", "together", "ollama"),
  model,
  trait = "overall_quality",
  trait_name = NULL,
  trait_description = NULL,
  prompt_template = set_prompt_template(),
  endpoint = "chat.completions",
  api_key = NULL,
  include_raw = FALSE,
  text_col = "text",
  judge_args = list()
)

Arguments

Details

The returned function has signature judge(A, B, state, ...) and enforces the adaptive transactional contract: it returns is_valid = TRUE with Y in ⁠{0,1}⁠ when the model response identifies one of the two presented items, and returns is_valid = FALSE otherwise. In addition to the required contract fields, the returned judge preserves canonical audit metadata from the first llm_compare_pair() row, including backend/model provenance, status/error fields, token counts, and a serialized raw_response_json payload when available.

Model configuration is split into:

• fixed build-time options via judge_args,

• per-run overrides via judge_call_args in adaptive_rank(),

• optional per-step overrides via ... passed through adaptive_rank_run_live().

Collectively this supports all llm_compare_pair() options, including backend-specific parameters such as OpenAI reasoning and service_tier.

Value

A function judge(A, B, state, ...) returning a list with required fields is_valid, Y, and invalid_reason, plus optional canonical audit fields such as judge_backend, judge_model, judge_endpoint, llm_status_code, llm_error_message, llm_custom_id, prompt_tokens, completion_tokens, total_tokens, and raw_response_json.

See Also

adaptive_rank(), adaptive_rank_run_live(), llm_compare_pair()

Other adaptive ranking: adaptive_rank(), adaptive_rank_resume(), adaptive_rank_run_live(), adaptive_rank_start(), summarize_adaptive()

Examples

judge <- make_adaptive_judge_llm(
  backend = "openai",
  model = "gpt-5.1",
  endpoint = "responses",
  judge_args = list(
    reasoning = "low",
    service_tier = "flex",
    include_thoughts = FALSE
  )
)

Create all unordered pairs of writing samples

Description

Given a data frame of samples with columns ID and text, this function generates all unordered pairs (combinations) of samples. Each pair appears exactly once, with ID1 < ID2 in lexicographic order.

Usage

make_pairs(samples)

Arguments

Value

A tibble with columns:

• ID1, text1

• ID2, text2

See Also

read_samples_df(), read_samples_dir()

Other pairing and data: alternate_pair_order(), randomize_pair_order(), read_samples_df(), read_samples_dir(), sample_pairs(), sample_reverse_pairs()

Examples

samples <- tibble::tibble(
  ID   = c("S1", "S2", "S3"),
  text = c("Sample 1", "Sample 2", "Sample 3")
)

pairs_all <- make_pairs(samples)
pairs_all

# Using the built-in example data
data("example_writing_samples")
pairs_example <- make_pairs(example_writing_samples)
nrow(pairs_example) # should be choose(10, 2) = 45

Live Ollama comparison for a single pair of samples

Description

ollama_compare_pair_live() sends a single pairwise comparison prompt to a local Ollama server and parses the result into the standard pairwiseLLM tibble format.

Usage

ollama_compare_pair_live(
  ID1,
  text1,
  ID2,
  text2,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  host = getOption("pairwiseLLM.ollama_host", "http://127.0.0.1:11434"),
  tag_prefix = "<BETTER_SAMPLE>",
  tag_suffix = "</BETTER_SAMPLE>",
  think = FALSE,
  num_ctx = 8192L,
  include_raw = FALSE,
  ...
)

Arguments

Details

The function targets the ⁠/api/generate⁠ endpoint on a running Ollama instance and expects a single non-streaming response. Model names should match those available in your Ollama installation (for example "mistral-small3.2:24b", "qwen3:32b", "gemma3:27b").

Sampling and context length are controlled as follows:

• Sampling parameters are omitted so the local model default applies.

• The context window is set via options$num_ctx, which defaults to 8192L but may be overridden via the num_ctx argument.

If the Ollama response includes a thinking field (as described in the Ollama API), that string is stored in the thoughts column of the returned tibble; otherwise thoughts is NA. This allows pairwiseLLM to consume Ollama's native thinking output in a way that is consistent with other backends that expose explicit reasoning traces.

The Ollama backend is intended to be compatible with the existing OpenAI, Anthropic, and Gemini backends, so the returned tibble can be used directly with downstream helpers such as build_bt_data() and fit_bt_model().

In typical workflows, users will call llm_compare_pair() with backend = "ollama" rather than using ollama_compare_pair_live() directly. The direct helper is exported so that advanced users can work with Ollama in a more explicit and backend-specific way.

The function assumes that:

• An Ollama server is running and reachable at host.

• The requested model has already been pulled, for example via ollama pull mistral-small3.2:24b on the command line.

When the Ollama response includes a thinking field (as documented in the Ollama API), that string is copied into the thoughts column of the returned tibble; otherwise thoughts is NA. This parsed thinking output can be logged, inspected, or analyzed alongside the visible comparison decisions.

Value

A tibble with one row and columns:

• custom_id – stable ID for the pair (pair_uid if supplied via ...; otherwise "LIVE_<ID1>_vs_<ID2>").

• ID1, ID2 – the sample IDs supplied to the function.

• model – model name reported by the API (or the requested model).

• object_type – backend object type (for example "ollama.generate").

• status_code – HTTP-style status code (200 if successful).

• error_message – error message if something goes wrong; otherwise NA.

• thoughts – reasoning / thinking text when a thinking field is returned by Ollama; otherwise NA.

• content – visible response text from the model (from the response field).

• better_sample – "SAMPLE_1", "SAMPLE_2", or NA, based on tags found in content.

• better_id – ID1 if "SAMPLE_1" is chosen, ID2 if "SAMPLE_2" is chosen, otherwise NA.

• prompt_tokens – prompt / input token count (if reported).

• completion_tokens – completion / output token count (if reported).

• total_tokens – total token count (if reported).

• raw_response – optional list-column containing the parsed JSON body (present only when include_raw = TRUE).

See Also

• submit_ollama_pairs_live() for single-backend, row-wise comparisons.

• llm_compare_pair() for backend-agnostic single-pair comparisons.

• submit_llm_pairs() for backend-agnostic comparisons over tibbles of pairs.

Other live backends: anthropic_compare_pair_live(), check_llm_api_keys(), gemini_compare_pair_live(), llm_compare_pair(), openai_compare_pair_live(), submit_anthropic_pairs_live(), submit_gemini_pairs_live(), submit_llm_pairs(), submit_ollama_pairs_live(), submit_openai_pairs_live(), submit_together_pairs_live(), submit_vertex_pairs_live(), together_compare_pair_live(), vertex_compare_pair_live()

Examples

## Not run: 
# Requires a running Ollama server and locally available models.

data("example_writing_samples", package = "pairwiseLLM")

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

ID1 <- example_writing_samples$ID[1]
ID2 <- example_writing_samples$ID[2]
text1 <- example_writing_samples$text[1]
text2 <- example_writing_samples$text[2]

# Make sure an Ollama server is running

# mistral example
res_mistral <- ollama_compare_pair_live(
  ID1               = ID1,
  text1             = text1,
  ID2               = ID2,
  text2             = text2,
  model             = "mistral-small3.2:24b",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl
)

res_mistral$better_id

# qwen example with reasoning
res_qwen_think <- ollama_compare_pair_live(
  ID1               = ID1,
  text1             = text1,
  ID2               = ID2,
  text2             = text2,
  model             = "qwen3:32b",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  think             = TRUE,
  include_raw       = TRUE
)

res_qwen_think$better_id
res_qwen_think$thoughts

## End(Not run)

Live OpenAI comparison for a single pair of samples

Description

This function sends a single pairwise comparison prompt to the OpenAI API and parses the result into a small tibble. It is the live / on-demand analogue of build_openai_batch_requests plus parse_openai_batch_output.

Usage

openai_compare_pair_live(
  ID1,
  text1,
  ID2,
  text2,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  endpoint = c("chat.completions", "responses"),
  tag_prefix = "<BETTER_SAMPLE>",
  tag_suffix = "</BETTER_SAMPLE>",
  api_key = NULL,
  include_raw = FALSE,
  ...
)

Arguments

Details

It supports both the Chat Completions endpoint ("/v1/chat/completions") and the Responses endpoint ("/v1/responses", for example gpt-5.6 with reasoning), using the same prompt template and model / parameter rules as the batch pipeline.

For the Responses endpoint, the function collects:

• Reasoning / "thoughts" text (if available) into the thoughts column.

• Visible assistant output into the content column.

Sampling defaults: If temperature or top_p is not provided in ..., the corresponding field is omitted so the model/provider default applies. Reasoning modes that do not support sampling parameters continue to require them to be NULL.

Value

A tibble with one row and columns:

custom_id

Stable ID for the pair (pair_uid if supplied via ...; otherwise "LIVE_<ID1>_vs_<ID2>").

ID1, ID2

The sample IDs you supplied.

model

Model name reported by the API.

object_type

OpenAI object type (for example "chat.completion" or "response").

status_code

HTTP-style status code (200 if successful).

error_message

Error message if something goes wrong; otherwise NA.

thoughts

Reasoning / thinking summary text when available, otherwise NA.

content

Concatenated text from the assistant's visible output. For the Responses endpoint this is taken from the type = "message" output items and does not include reasoning summaries.

better_sample

"SAMPLE_1", "SAMPLE_2", or NA.

better_id

ID1 if SAMPLE_1 is chosen, ID2 if SAMPLE_2 is chosen, otherwise NA.

prompt_tokens

Prompt / input token count (if reported).

completion_tokens

Completion / output token count (if reported).

total_tokens

Total token count (if reported).

raw_response

(Optional) list-column containing the parsed JSON body.

See Also

check_llm_api_keys(), llm_compare_pair()

Other live backends: anthropic_compare_pair_live(), check_llm_api_keys(), gemini_compare_pair_live(), llm_compare_pair(), ollama_compare_pair_live(), submit_anthropic_pairs_live(), submit_gemini_pairs_live(), submit_llm_pairs(), submit_ollama_pairs_live(), submit_openai_pairs_live(), submit_together_pairs_live(), submit_vertex_pairs_live(), together_compare_pair_live(), vertex_compare_pair_live()

Examples

## Not run: 
# Requires API key set and internet access

# 1. Standard comparison using GPT-4.1
res <- openai_compare_pair_live(
  ID1 = "A", text1 = "Text A...",
  ID2 = "B", text2 = "Text B...",
  model = "gpt-4.1",
  trait_name = "clarity",
  trait_description = "Which text is clearer?"
)

# 2. Reasoning comparison using GPT-5.6 Sol
res_reasoning <- openai_compare_pair_live(
  ID1 = "A", text1 = "Text A...",
  ID2 = "B", text2 = "Text B...",
  model = "gpt-5.6-sol",
  trait_name = "clarity",
  trait_description = "Which text is clearer?",
  endpoint = "responses",
  include_thoughts = TRUE,
  reasoning = "high",
  service_tier = "flex"
)
print(res_reasoning$thoughts)

## End(Not run)

Create an OpenAI batch from an uploaded file

Description

Creates and executes a batch based on a previously uploaded input file.

Usage

openai_create_batch(
  input_file_id,
  endpoint,
  completion_window = "24h",
  metadata = NULL,
  api_key = NULL
)

Arguments

Value

A list representing the Batch object.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

## Not run: 
# Requires OPENAI_API_KEY set in your environment and network access.

file_obj <- openai_upload_batch_file("batch_input.jsonl")

batch_obj <- openai_create_batch(
  input_file_id = file_obj$id,
  endpoint      = "/v1/chat/completions"
)

batch_obj$status

## End(Not run)

Download the output file for a completed batch

Description

Given a batch ID, retrieves the batch metadata, extracts the output_file_id, and downloads the corresponding file content to path.

Usage

openai_download_batch_output(batch_id, path, api_key = NULL)

Arguments

Value

Invisibly, the path to the downloaded file.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

## Not run: 
# Requires OPENAI_API_KEY and a completed batch with an output_file_id.

openai_download_batch_output("batch_abc123", "batch_output.jsonl")

# You can then parse the file
res <- parse_openai_batch_output("batch_output.jsonl")
head(res)

## End(Not run)

Retrieve an OpenAI batch

Description

Retrieve an OpenAI batch

Usage

openai_get_batch(batch_id, api_key = NULL)

Arguments

Value

A list representing the Batch object.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_output(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()