Results
results
CalculatedCostWithDimensions
Bases: dict
Result of a cost estimate (composed of multiple cost dimensions)
This class is a dictionary of costs keyed by dimension name, but provides additional convenience methods, including:
- A
.totalproperty, providing the total cost across all dimensions - Ability to add multiple cost estimates together (using standard
+orsum()) - Functions to save the estimate to (or load one from) a namespace including other information
(such as another dictionary, an
InvocationResponseorResult)
total
property
total
The total/overall cost over all dimensions
__add__
__add__(other)
Add two cost estimates together, or add a dimension to a cost estimate
Dimensions present by name in both self and other will be summed together.
Source code in llmeter/callbacks/cost/results.py
33 34 35 36 37 38 39 40 41 42 43 44 45 | |
__radd__
__radd__(other)
Our add operator is commutative, so radd just calls add
This definition supports sum([...costs]), which starts with 0 + costs[0] and therefore
needs costs[0] to support __radd__.
Source code in llmeter/callbacks/cost/results.py
47 48 49 50 51 52 53 54 55 | |
load_from_namespace
classmethod
load_from_namespace(raw, key_prefix='', ignore_total=True)
Load a CalculatedCostWithDimensions from a (potentially shared) namespace/object
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw
|
object
|
The namespace (dataclass, dictionary, etc) containing cost data |
required |
key_prefix
|
str
|
If specified, only keys starting with this prefix will be included, and the prefix will be removed from the generated cost dimension names. Use this for loading cost results from classes like InvocationResponse or Result, which may contain other information. |
''
|
ignore_total
|
bool
|
If True [default], the |
True
|
Source code in llmeter/callbacks/cost/results.py
128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 | |
merge
merge(other)
Merge another cost estimate into this one (in-place)
Dimensions present in both self and other will be summed together.
Source code in llmeter/callbacks/cost/results.py
57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | |
save_on_namespace
save_on_namespace(raw, key_prefix='', include_total=True)
Store this cost result on a (potentially shared) namespace/object
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw
|
object
|
The namespace (dataclass, dictionary, etc) containing cost data |
required |
key_prefix
|
str
|
If specified, only keys starting with this prefix will be included, and the prefix will be removed from the generated cost dimension names. Use this for loading cost results from classes like InvocationResponse or Result, which may contain other information. |
''
|
include_total
|
bool
|
If True [default], the |
True
|
Source code in llmeter/callbacks/cost/results.py
163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 | |
summary_statistics
staticmethod
summary_statistics(calculated_costs, key_prefix='', key_dim_name_suffix='', key_stat_name_prefix='-', key_total_name_and_suffix='total')
Utility function to calculate summary statistics for a dataset of cost results
At a high level, this method produces a flat map from keys like [dimension]-[statistic], to the value of that statistic for that dimension.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
calculated_costs
|
Sequence[CalculatedCostWithDimensions]
|
Sequence of CalculatedCostWithDimensions results to summarize |
required |
key_prefix
|
str
|
Prefix to add to each key in the output dictionary. This is useful in case
the output will be merged with other statistics. Set to |
''
|
key_dim_name_suffix
|
str
|
Suffix to add to each dimension name in the output dictionary.
Set to |
''
|
key_stat_name_prefix
|
str
|
Separator to use before the name of the statistic in output keys. |
'-'
|
key_total_name_and_suffix
|
str
|
Name to use for the |
'total'
|
Returns:
| Name | Type | Description |
|---|---|---|
stats |
dict[str, Number]
|
A flat dictionary of summary statistics from all dimensions of the input
|
Source code in llmeter/callbacks/cost/results.py
74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 | |