Skip to content

Tokenizers

tokenizers

DummyTokenizer 

DummyTokenizer(*args, **kwargs)

Bases: Tokenizer

A dummy tokenizer that splits the input text on whitespace and returns the tokens as is.

This tokenizer will generally under-estimate token counts in English and latin languages (where words comprise more than one token on average), and will give very poor results for languages where the whitespace/"word" heuristic doesn't work well (e.g. Chinese, Japanese, Korean, Thai).

However, it requires no dependencies beyond the Python standard library, using str.split()

Source code in llmeter/tokenizers.py 
98
99
def __init__(self, *args, **kwargs):
    pass

Tokenizer 

Tokenizer(*args, **kwargs)

Bases: Serializable, ABC

Source code in llmeter/tokenizers.py 
12
13
def __init__(self, *args, **kwargs):
    pass

load staticmethod 

load(tokenizer_info)

Load a tokenizer from a dictionary.

This supports configs saved before the unified dump_object/load_object serialization was introduced. New code should use :func:~llmeter.serialization.load_object instead.

Parameters:

Name Type Description Default
tokenizer_info dict

The tokenizer information to load. Must include at minimum a tokenizer_module key.

 required

Returns:

Name Type Description
Tokenizer 'Tokenizer'

The loaded tokenizer.
Source code in llmeter/tokenizers.py 
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
@staticmethod
def load(tokenizer_info: dict) -> "Tokenizer":
    """Load a tokenizer from a dictionary.

    This supports configs saved before the unified ``dump_object``/``load_object``
    serialization was introduced. New code should use
    :func:`~llmeter.serialization.load_object` instead.

    Args:
        tokenizer_info (dict): The tokenizer information to load. Must include at minimum
            a ``tokenizer_module`` key.

    Returns:
        Tokenizer: The loaded tokenizer.
    """
    return _load_tokenizer_from_info(tokenizer_info)