TiktokenCounter

TiktokenCounter estimates the token count of ChatMessage objects and optional tool schemas with OpenAI's tiktoken byte-pair encoder. It is generally more accurate than a character-based estimate for OpenAI models, but its results can differ from the token counts of other providers.

Import path	haystack.token_counters.TiktokenCounter
API reference	Token Counters
GitHub link	https://github.com/deepset-ai/haystack/blob/main/haystack/token_counters/tiktoken_counter.py
Package name	haystack-ai

Installation

Install the optional tiktoken dependency before constructing the counter:

bash

pip install tiktoken

Usage

Create the counter and pass a list of messages to count():

python

from haystack.dataclasses import ChatMessage
from haystack.token_counters import TiktokenCounter

messages = [
    ChatMessage.from_system("You are a helpful assistant."),
    ChatMessage.from_user("Explain retrieval-augmented generation."),
]

counter = TiktokenCounter()
token_count = counter.count(messages)
print(token_count)

The default encoding is o200k_base. Pass a different encoding when required by your model:

python

counter = TiktokenCounter(encoding="cl100k_base")

The counter loads its encoding on the first call to count(). To load it during application startup instead, call warm_up() explicitly:

python

counter.warm_up()

To include the context consumed by tool schemas, pass the tools to count():

python

token_count = counter.count(messages, tools=[search_tool])

Non-text content

The tokenizer cannot measure images or files, so the counter adds a flat estimate for each item. Change the defaults when your application sends large images or long documents:

python

counter = TiktokenCounter(
    encoding="o200k_base",
    tokens_per_image=765,
    tokens_per_file=4000,
)

The counter includes non-text content attached directly to a message as well as content nested inside tool results.