Mlflow

mlflow

MlflowCallback

MlflowCallback(step=None, nested=False)

Bases: Callback

LLMeter callback to log test run parameters and metrics to an MLFlow tracking server.

This callback integrates with MLflow to track and log parameters and metrics from LLMeter test runs. It can operate either in the current MLflow run context or create nested runs for each test.

Example

import mlflow
from llmeter.callbacks import MlflowCallback

with mlflow.start_run():
    runner = Runner(
        endpoint=endpoint,
        callbacks=[MlflowCallback()]
    )
    results = await runner.run()

Attributes:

Name	Type	Description
`step`	`int \| None`	Step number for MLflow metrics logging
`nested`	`bool`	Whether to create nested runs for each test
`parameters_names`	`list`	List of parameter names to log to MLflow

Parameters:

Name	Type	Description	Default
`step`	`int \| None`	Passed through to `mlflow.log_metrics`	`None`
`nested`	`bool`	By default (`False`) the externally-provided mlflow run context will be used. Set `True` to log under a new nested run with the name of the LLMeter `Result.run_name`.	`False`

Raises:

Type	Description
`ImportError`	If MLflow is not installed

Source code in llmeter/callbacks/mlflow.py

def __init__(self, step: int | None = None, nested: bool = False) -> None:
    """Create an MlflowCallback

    Args:
        step (int | None, optional): Passed through to `mlflow.log_metrics`
        nested (bool, optional): By default (`False`) the externally-provided mlflow run context will be used.
            Set `True` to log under a new nested run with the name of the LLMeter
            `Result.run_name`.

    Raises:
        ImportError: If MLflow is not installed
    """
    super().__init__()
    self.step = step
    self.nested = nested
    # Check MLFlow is installed by polling any attribute on the module to trigger DeferredError
    mlflow.__version__

after_run `async`

after_run(result)

Callback method executed after an LLMeter test run completes.

Logs parameters and metrics to MLflow, either in the current run context or in a new nested run depending on the nested setting.