A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
| Version: | 0.0.7 | 
| Depends: | R (≥ 4.2) | 
| Suggests: | testthat (≥ 3.0.0) | 
| Published: | 2025-04-14 | 
| DOI: | 10.32614/CRAN.package.rtiktoken | 
| Author: | David Zimmermann-Kollenda [aut, cre],
  Roger Zurawicki [aut] (tiktoken-rs Rust library),
  Authors of the dependent Rust crates [aut] (see AUTHORS file) rtiktoken author details | 
| Maintainer: | David Zimmermann-Kollenda <david_j_zimmermann at hotmail.com> | 
| BugReports: | https://github.com/DavZim/rtiktoken/issues | 
| License: | MIT + file LICENSE | 
| URL: | https://davzim.github.io/rtiktoken/, https://github.com/DavZim/rtiktoken/ | 
| NeedsCompilation: | yes | 
| SystemRequirements: | Cargo (Rust's package manager), rustc >= 1.65.0 | 
| Materials: | README, NEWS | 
| CRAN checks: | rtiktoken results | 
| Reference manual: | rtiktoken.html , rtiktoken.pdf | 
| Package source: | rtiktoken_0.0.7.tar.gz | 
| Windows binaries: | r-devel: rtiktoken_0.0.7.zip, r-release: rtiktoken_0.0.7.zip, r-oldrel: rtiktoken_0.0.7.zip | 
| macOS binaries: | r-release (arm64): rtiktoken_0.0.7.tgz, r-oldrel (arm64): rtiktoken_0.0.7.tgz, r-release (x86_64): rtiktoken_0.0.7.tgz, r-oldrel (x86_64): rtiktoken_0.0.7.tgz | 
| Old sources: | rtiktoken archive | 
Please use the canonical form https://CRAN.R-project.org/package=rtiktoken to link to this page.