Skip to content

Base

base

Base classes used across callbacks submodules

Callback

Bases: Serializable, ABC

Base class for a callback in LLMeter

Callbacks support extending LLMeter functionality by running additional code at defined points in the test Run lifecycle: For example, logging experiments to MLFlow, or estimating costs associated with test runs or individual model invocations.

A Callback object may implement multiple of the defined lifecycle hooks (such as before_invoke, after_run, etc) - which have no-op implementations by default. Serialization to/from file is inherited from llmeter.serialization.Serializable, and is necessary so your callback(s) can be saved to file (and restored) as part of a Run configuration. Any custom callback class that is not LLMeter-serializable will raise a TypeError when the run config it belongs to is saved.

after_invoke async 

after_invoke(response)

Lifecycle hook called after every Endpoint.invoke() request in a Run.

Parameters:

Name Type Description Default
response InvocationResponse

The InvocationResponse (already annotated with initial information e.g. timing and token counts)

 required

Returns: None: If you'd like to add information to the response logged in the Run, modify it in-place. To attach extra custom fields that you want preserved when responses are saved to file, store them in the response.annotations map.

Source code in llmeter/callbacks/base.py 
40
41
42
43
44
45
46
47
48
49
50
51
async def after_invoke(self, response: InvocationResponse) -> None:
    """Lifecycle hook called after every `Endpoint.invoke()` request in a Run.

    Args:
        response: The InvocationResponse (already annotated with initial information e.g.
            timing and token counts)
    Returns:
        None: If you'd like to add information to the `response` logged in the Run, modify it
            in-place. To attach **extra custom fields** that you want preserved when responses
            are saved to file, store them in the `response.annotations` map.
    """
    pass

after_run async 

after_run(result)

Lifecycle hook called at the end of each Runner.run()

Parameters:

Name Type Description Default
result Result

The Result of the overall run (including all individual model invocations)

 required

Returns: None: If you'd like to modify the run result, edit the argument in-place.

Source code in llmeter/callbacks/base.py 
66
67
68
69
70
71
72
73
74
async def after_run(self, result: Result) -> None:
    """Lifecycle hook called at the end of each `Runner.run()`

    Args:
        result: The Result of the overall run (including all individual model invocations)
    Returns:
        None: If you'd like to modify the run `result`, edit the argument in-place.
    """
    pass

before_invoke async 

before_invoke(payload)

Lifecycle hook called before every Endpoint.invoke() request in a Run.

Parameters:

Name Type Description Default
payload dict

The payload to be sent to the endpoint.

 required

Returns: None: If you'd like to modify the request payload, edit the dictionary in-place.

Source code in llmeter/callbacks/base.py 
30
31
32
33
34
35
36
37
38
async def before_invoke(self, payload: dict) -> None:
    """Lifecycle hook called before every `Endpoint.invoke()` request in a Run.

    Args:
        payload: The payload to be sent to the endpoint.
    Returns:
        None: If you'd like to modify the request `payload`, edit the dictionary in-place.
    """
    pass

before_run async 

before_run(run_config)

Lifecycle hook called at the start of each Runner.run()

This function will be called after the initial Runner configuration is prepared, and before creating clients or starting to send requests.

Parameters:

Name Type Description Default
run_config _RunConfig

The configuration that will be used to run the test.

 required

Returns: None: If you'd like to modify the current run's configuration, edit it in-place.

Source code in llmeter/callbacks/base.py 
53
54
55
56
57
58
59
60
61
62
63
64
async def before_run(self, run_config: _RunConfig) -> None:
    """Lifecycle hook called at the start of each `Runner.run()`

    This function will be called after the initial Runner configuration is prepared, and before
    creating clients or starting to send requests.

    Args:
        run_config: The configuration that will be used to run the test.
    Returns:
        None: If you'd like to modify the current run's configuration, edit it in-place.
    """
    pass