R: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models

# A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models

## Documentation for package ‘rtiktoken’ version 0.0.7

- DESCRIPTION file
- Package NEWS
- Package README

## Help Pages

decode_tokens | Decodes tokens back to text
get_tokens | Converts text to tokens
get_token_count | Returns the number of tokens in a text
model_to_tokenizer | Gets the name of the tokenizer used by a model