Skip to content

Results

results

CalculatedCostWithDimensions

Bases: dict

Result of a cost estimate (composed of multiple cost dimensions)

This class is a dictionary of costs keyed by dimension name, but provides additional convenience methods, including:

  • A.total property, providing the total cost across all dimensions
  • Ability to add multiple cost estimates together (using standard + or sum())
  • Functions to save the estimate to (or load one from) a namespace including other information (such as another dictionary, an InvocationResponse or Result)

total property

total

The total/overall cost over all dimensions

__add__

__add__(other)

Add two cost estimates together, or add a dimension to a cost estimate

Dimensions present by name in both self and other will be summed together.

Source code in llmeter/callbacks/cost/results.py
33
34
35
36
37
38
39
40
41
42
43
44
45
def __add__(
    self, other: dict[str, Number] | CalculatedCostWithDimensions | Literal[0]
) -> CalculatedCostWithDimensions:
    """Add two cost estimates together, or add a dimension to a cost estimate

    Dimensions present by name in both `self` and `other` will be summed together.
    """
    result = CalculatedCostWithDimensions()  # Work from a copy
    result.merge(self)
    if other == 0:  # Important to handle this case for sum()
        return result
    result.merge(other)
    return result

__radd__

__radd__(other)

Our add operator is commutative, so radd just calls add

This definition supports sum([...costs]), which starts with 0 + costs[0] and therefore needs costs[0] to support __radd__.

Source code in llmeter/callbacks/cost/results.py
47
48
49
50
51
52
53
54
55
def __radd__(
    self, other: dict[str, Number] | CalculatedCostWithDimensions | Literal[0]
) -> CalculatedCostWithDimensions:
    """Our __add__ operator is commutative, so __radd__ just calls __add__

    This definition supports `sum([...costs])`, which starts with `0 + costs[0]` and therefore
    needs `costs[0]` to support `__radd__`.
    """
    return self.__add__(other)

load_from_namespace classmethod

load_from_namespace(raw, key_prefix='', ignore_total=True)

Load a CalculatedCostWithDimensions from a (potentially shared) namespace/object

Parameters:

Name Type Description Default
raw object

The namespace (dataclass, dictionary, etc) containing cost data

required
key_prefix str

If specified, only keys starting with this prefix will be included, and the prefix will be removed from the generated cost dimension names. Use this for loading cost results from classes like InvocationResponse or Result, which may contain other information.

''
ignore_total bool

If True [default], the {key_prefix}total key will be ignored if present. This is useful when working with results saved via save_on_namespace() with include_total=True.

True
Source code in llmeter/callbacks/cost/results.py
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
@classmethod
def load_from_namespace(
    cls,
    raw: object,
    key_prefix: str = "",
    ignore_total: bool = True,
) -> CalculatedCostWithDimensions:
    """Load a `CalculatedCostWithDimensions` from a (potentially shared) namespace/object

    Args:
        raw: The namespace (dataclass, dictionary, etc) containing cost data
        key_prefix: If specified, only keys starting with this prefix will be included, and the
            prefix will be removed from the generated cost dimension names. Use this for
            loading cost results from classes like InvocationResponse or Result, which may
            contain other information.
        ignore_total: If True [default], the `{key_prefix}total` key will be ignored if
            present. This is useful when working with results saved via `save_on_namespace()`
            with `include_total=True`.
    """
    if hasattr(raw, "__dict__"):
        dict_args = raw.__dict__
    elif hasattr(raw, "items") and callable(raw.items):
        dict_args = {k: v for k, v in raw.items()}
    else:
        dict_args = {k: getattr(raw, k) for k in dir(raw)}
    if key_prefix:
        dict_args = {
            k[len(key_prefix) :]: v
            for k, v in dict_args.items()
            if k.startswith(key_prefix)
        }
    if ignore_total:
        dict_args.pop("total", None)
    return cls(**dict_args)

merge

merge(other)

Merge another cost estimate into this one (in-place)

Dimensions present in both self and other will be summed together.

Source code in llmeter/callbacks/cost/results.py
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
def merge(self, other: dict[str, Number] | CalculatedCostWithDimensions) -> None:
    """Merge another cost estimate into this one (in-place)

    Dimensions present in both `self` and `other` will be summed together.
    """
    if not isinstance(other, (CalculatedCostWithDimensions, dict)):
        raise TypeError(
            "Can only merge a CalculatedCostWithDimensions to a dictionary or another "
            "CalculatedCostWithDimensions (got %s)" % (other,)
        )
    # Merge dimensions from other:
    for k, v in other.items():
        if k in self:
            self[k] += v
        else:
            self[k] = v

save_on_namespace

save_on_namespace(raw, key_prefix='', include_total=True)

Store this cost result on a (potentially shared) namespace/object

Parameters:

Name Type Description Default
raw object

The namespace (dataclass, dictionary, etc) containing cost data

required
key_prefix str

If specified, only keys starting with this prefix will be included, and the prefix will be removed from the generated cost dimension names. Use this for loading cost results from classes like InvocationResponse or Result, which may contain other information.

''
include_total bool

If True [default], the .total property will also be written to the {key_prefix}total key on the output. Set False to omit.

True
Source code in llmeter/callbacks/cost/results.py
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
def save_on_namespace(
    self,
    raw: object,
    key_prefix: str = "",
    include_total: bool = True,
) -> None:
    """Store this cost result on a (potentially shared) namespace/object

    Args:
        raw: The namespace (dataclass, dictionary, etc) containing cost data
        key_prefix: If specified, only keys starting with this prefix will be included, and the
            prefix will be removed from the generated cost dimension names. Use this for
            loading cost results from classes like InvocationResponse or Result, which may
            contain other information.
        include_total: If True [default], the `.total` property will also be written to the
            `{key_prefix}total` key on the output. Set False to omit.
    """

    def setter(target, key, value):
        if hasattr(target, "__setitem__"):
            target[key] = value
        else:
            setattr(target, key, value)

    for name, value in self.items():
        setter(raw, f"{key_prefix}{name}", value)
    if include_total:
        setter(raw, f"{key_prefix}total", self.total)

summary_statistics staticmethod

summary_statistics(calculated_costs, key_prefix='', key_dim_name_suffix='', key_stat_name_prefix='-', key_total_name_and_suffix='total')

Utility function to calculate summary statistics for a dataset of cost results

At a high level, this method produces a flat map from keys like [dimension]-[statistic], to the value of that statistic for that dimension.

Parameters:

Name Type Description Default
calculated_costs Sequence[CalculatedCostWithDimensions]

Sequence of CalculatedCostWithDimensions results to summarize

required
key_prefix str

Prefix to add to each key in the output dictionary. This is useful in case the output will be merged with other statistics. Set to "cost_" to match CostModel callback's default treatment, or leave default for no prefix.

''
key_dim_name_suffix str

Suffix to add to each dimension name in the output dictionary. Set to "_per_request" to match CostModel callback's default treatment, or leave default for no suffix.

''
key_stat_name_prefix str

Separator to use before the name of the statistic in output keys.

'-'
key_total_name_and_suffix str

Name to use for the .total dimension, and its attached suffix if one should be present. Set to "per_request" to match CostModel callback's default treatment, or leave default to call the dimension "total".

'total'

Returns:

Name Type Description
stats dict[str, Number]

A flat dictionary of summary statistics from all dimensions of the input calculated_costs, and their totals.

Source code in llmeter/callbacks/cost/results.py
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
@staticmethod
def summary_statistics(
    calculated_costs: Sequence[CalculatedCostWithDimensions],
    key_prefix: str = "",
    key_dim_name_suffix: str = "",
    key_stat_name_prefix: str = "-",
    key_total_name_and_suffix: str = "total",
) -> dict[str, Number]:
    """Utility function to calculate summary statistics for a dataset of cost results

    At a high level, this method produces a flat map from keys like [dimension]-[statistic], to
    the value of that statistic for that dimension.

    Args:
        calculated_costs: Sequence of CalculatedCostWithDimensions results to summarize
        key_prefix: Prefix to add to each key in the output dictionary. This is useful in case
            the output will be merged with other statistics. Set to `"cost_"` to match
            CostModel callback's default treatment, or leave default for no prefix.
        key_dim_name_suffix: Suffix to add to each dimension name in the output dictionary.
            Set to `"_per_request"` to match CostModel callback's default treatment, or leave
            default for no suffix.
        key_stat_name_prefix: Separator to use before the name of the statistic in output keys.
        key_total_name_and_suffix: Name to use for the `.total` dimension, *and* its attached
            suffix if one should be present. Set to `"per_request"` to match CostModel
            callback's default treatment, or leave default to call the dimension "total".

    Returns:
        stats: A flat dictionary of summary statistics from all dimensions of the input
            `calculated_costs`, and their totals.
    """
    vals_by_dimension: dict[str, list[Number]] = {}
    total_vals: list[Number] = []
    for c in calculated_costs:
        for dim_name, dim_cost in c.items():
            if dim_name not in vals_by_dimension:
                vals_by_dimension[dim_name] = []
            vals_by_dimension[dim_name].append(dim_cost)
        total_vals.append(c.total)
    # Dimension-level stats:
    result = {
        f"{key_prefix}{dim_name}{key_dim_name_suffix}{key_stat_name_prefix}{stat_name}": stat_val
        for dim_name, dim_values in vals_by_dimension.items()
        if isinstance(dim_values[0], Number)
        for stat_name, stat_val in summary_stats_from_list(dim_values).items()
    }
    # Total stats:
    result.update(
        {
            f"{key_prefix}{key_total_name_and_suffix}{key_stat_name_prefix}{stat_name}": stat_val
            for stat_name, stat_val in summary_stats_from_list(total_vals).items()
        }
    )
    return result