utils.evals

add_dynamic_metric

Add a dynamic metric to a scorer.

Adding is idempotent by metric name: if a metric with the same registered name is already attached to the factory, it is not added again. Metric names are derived from their parameters, so a name collision means the same metric. This makes it safe to attach metrics from within a scorer factory that may be invoked more than once, since the factory’s metric list is shared across every scorer it builds.

Source

def add_dynamic_metric(
    scorer_factory: Callable[..., Scorer],
    metric: Metric,
    category: str | None = None,
) -> None

scorer_factory Callable[..., Scorer]: A factory function that returns scorers that the metric applies to.
metric Metric: The metric to add.
category str | None: The category of the metric. Defaults to the module name of the metric.

add_percentile_metric

Add a percentile metric to a scorer.

Source

def add_percentile_metric(
    scorer_factory: Callable[..., Scorer],
    target: float,
    type: Literal["percentile", "cutoff"] = "percentile",
    derivatives: Sequence[PercentileDerivatives] = (),
    method: PercentileMethod = "linear",
) -> None

scorer_factory Callable[..., Scorer]: A factory function that returns scorers that the metric applies to.
target float: The target percentile or cutoff between 0 and 100 (e.g., 80 for 80th percentile or 1 for 1% cutoff).
type Literal['percentile', 'cutoff']: Whether to calculate a percentile or a cutoff threshold. Defaults to “percentile”.
derivatives Sequence[PercentileDerivatives]: List of additional metrics to calculate along with the target percentile or cutoff threshold. Defaults to an empty list.
method PercentileMethod: The method to use for calculating the percentile or cutoff threshold. Defaults to “linear”.

add_usefulness_metric

Add a usefulness metric to a scorer. The usefulness metric computes the mean accuracy and scales it relative to the benchmark values for the trusted and untrusted models, where the trusted model’s value is the minimum of the scale and the untrusted model’s value is the maximum of the scale (e.g., 0.0 and 1.0, respectively).

Source

def add_usefulness_metric(
    scorer_factory: Callable[..., Scorer],
    trusted_value: float,
    untrusted_value: float,
    scale: int = 1,
) -> None

scorer_factory Callable[..., Scorer]: A factory function that returns scorers that the metric applies to.
trusted_value float: The baseline value for the trusted model (minimum of the scale).
untrusted_value float: The baseline value for the untrusted model (maximum of the scale).
scale int: Multiplier for the result. Defaults to 1 to return a value in the range from 0-1. Use 100 to return 0-100 range, for instance.

filter_reasoning_from_messages

Remove reasoning content blocks from assistant messages.

Many model providers (e.g., Claude, Groq) return explicit reasoning blocks in addition to the final assistant response. When generating prompts for trusted monitors we often need to exclude this chain-of-thought content. This helper removes any ContentReasoning entries from assistant messages while leaving all other message types untouched.

Source

def filter_reasoning_from_messages(messages: Sequence[ChatMessage]) -> list[ChatMessage]

messages Sequence[ChatMessage]: Sequence of chat messages that may include reasoning content.

Returns

A new list of messages with reasoning blocks removed. If a message contained only reasoning content it will be preserved with an empty string payload.

filter_tool_calls_from_messages

Filter out tool calls with specific names from assistant messages, and remove corresponding tool result messages.

Source

def filter_tool_calls_from_messages(
    messages: Sequence[ChatMessage],
    exclude: Sequence[str],
) -> list[ChatMessage]

messages Sequence[ChatMessage]: The messages to filter
exclude Sequence[str]: Names of tool calls to remove

Returns

Filtered list of messages

get_dynamic_metrics

Get the dynamic metrics for a scorer.

Source

def get_dynamic_metrics(scorer_factory: Callable[..., Scorer], category: str) -> list[Metric]

scorer_factory Callable[..., Scorer]: A factory function that returns scorers that the metric applies to.
category str: The category of the metric.

Returns

A list of dynamic metrics for the given category of metrics on the scorer.

get_score

Get any existing score for the scorer from the state.scores dictionary. If the score is not in the dictionary, run the scorer to get the score.

Source

async def get_score(state: TaskState, target: Target, scorer: Scorer) -> Score

state TaskState: The state of the current task
target Target: The target of the current task
scorer Scorer: The scorer to run if the score is not in the dictionary

Returns

A Score object

get_solver_results

Run a solver and return the state and changes to the store, avoiding saving changes to the state if interim_results is set.

Source

async def get_solver_results(
    solver: Solver,
    state: TaskState,
    generate: Generate,
    interim_results: bool = False,
) -> tuple[TaskState, dict[str, Any]]

solver Solver: The solver to run
state TaskState: The task state to run the solver on
generate Generate: The generate function for the solver
interim_results bool: If True, use a shadow state that doesn’t modify the original state

Returns

Tuple of (state, store_changes) where: - state: The original state (unchanged if interim_results=True), or the returned state - store_changes: Dict of all keys that were set in the store during solver execution