API Reference

Protocols

CheckpointStoreProtocol

source

Digest-verified persistence of run checkpoints.

save

async def save(
    run_id: str,
    slug: str,
    payload: dict[str, Any]
) -> Checkpoint

source

Persist a checkpoint for a run.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.
`slug`	str	Stable name of the checkpoint within the run.
`payload`	dict[str, Any]	State to checkpoint.

Returns

Type	Description
Checkpoint	The stored checkpoint with its content digest.

load

async def load(
    run_id: str,
    slug: str
) -> Checkpoint | None

source

Load a checkpoint, verifying its content digest.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.
`slug`	str	Checkpoint name.

Returns

Type	Description
Checkpoint | None	The verified checkpoint, or ``None`` when absent or tampered.

list

async def list(run_id: str) -> list[Checkpoint]

source

List all checkpoints for a run.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.

Returns

Type	Description
list[Checkpoint]	Checkpoints in creation order.

---

ExperimentTrackerProtocol

source

Persistent, seed-stable tracking of experiment runs.

Implementations must derive a stable run id from the experiment config so rerunning the same seed and knobs resumes (or reproduces) the same run.

start

async def start(config: ExperimentConfig) -> ExperimentRun

source

Start (or resume) an experiment run for the given config.

Parameters

Parameter	Type	Description
`config`	ExperimentConfig	Seed and knob configuration of the run.

Returns

Type	Description
ExperimentRun	The started or resumed run.

log_metric

async def log_metric(
    run_id: str,
    name: str,
    value: float,
    step: int = 0
) -> None

source

Record a scalar metric for a run.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.
`name`	str	Metric name.
`value`	float	Metric value.
`step`	int	Step index the metric was recorded at. Defaults to 0.

log_error

async def log_error(
    run_id: str,
    kind: str,
    message: str,
    step: int = 0
) -> None

source

Record an error for a run.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.
`kind`	str	Error kind or code.
`message`	str	Human-readable error message.
`step`	int	Step index the error occurred at. Defaults to 0.

metrics

async def metrics(run_id: str) -> list[MetricRecord]

source

Return all metric records for a run.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.

Returns

Type	Description
list[MetricRecord]	Metric records in recording order.

errors

async def errors(run_id: str) -> list[ErrorRecord]

source

Return all error records for a run.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.

Returns

Type	Description
list[ErrorRecord]	Error records in recording order.

snapshot

async def snapshot(run_id: str) -> dict[str, Any]

source

Return a compact summary of a run’s current state.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.

Returns

Type	Description
dict[str, Any]	Latest value per metric, error counts, and run metadata.

resume

async def resume(run_id: str) -> ExperimentRun | None

source

Return an already-started run, or None when unknown.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.

Returns

Type	Description
ExperimentRun | None	The existing run, or ``None``.

finish

async def finish(
    run_id: str,
    status: RunStatus = RunStatus.COMPLETED
) -> None

source

Mark a run finished.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.
`status`	RunStatus	Terminal status. Defaults to ``COMPLETED``.

---

Classes

AblationResult

source

Delta comparison between a baseline and an ablated checkpoint.

Attributes: run_id: Run the ablation was performed on. knob: The configuration knob that was ablated. baseline_slug: Checkpoint slug of the baseline run. ablated_slug: Checkpoint slug of the ablated run. deltas: Per-metric differences (ablated minus baseline). digest: SHA-256 digest of the ablation inputs and deltas.

---

AblationRunner

source

Compare baseline and ablated checkpoints from a checkpoint store.

Parameters

Parameter	Type	Description
`store`		Checkpoint store holding the baseline and ablated payloads.

__init__

def __init__(store: CheckpointStoreProtocol) -> None

source

deltas

def deltas(
    before: dict[str, float],
    after: dict[str, float]
) -> dict[str, float]

source

Compute per-metric deltas (after minus before).

Parameters

Parameter	Type	Description
`before`	dict[str, float]	Baseline metrics (name to value).
`after`	dict[str, float]	Ablated metrics (name to value).

Returns

Type	Description
dict[str, float]	Metric name to delta mapping, including keys from either side.

run

async def run(
    run_id: str,
    knob: str,
    baseline_slug: str,
    ablated_slug: str
) -> Result[AblationResult, AblationError]

source

Compare a baseline checkpoint against an ablated one in one run.

Parameters

Parameter	Type	Description
`run_id`	str	Run the ablation was performed on.
`knob`	str	The configuration knob that was ablated.
`baseline_slug`	str	Checkpoint slug of the baseline run.
`ablated_slug`	str	Checkpoint slug of the ablated run.

Returns

Type	Description
Result[AblationResult, AblationError]	Ok(AblationResult) with per-metric deltas, or Err(AblationError) when either checkpoint is missing.

compare

async def compare(
    knob: str,
    baseline_run_id: str,
    baseline_slug: str,
    ablated_run_id: str,
    ablated_slug: str
) -> Result[AblationResult, AblationError]

source

Compare checkpoints across two runs (e.g. control vs ablated).

Parameters

Parameter	Type	Description
`knob`	str	The configuration knob that was ablated.
`baseline_run_id`	str	Run identifier of the baseline.
`baseline_slug`	str	Checkpoint slug of the baseline run.
`ablated_run_id`	str	Run identifier of the ablated run.
`ablated_slug`	str	Checkpoint slug of the ablated run.

Returns

Type	Description
Result[AblationResult, AblationError]	Ok(AblationResult) with per-metric deltas, or Err(AblationError) when either checkpoint is missing.

Example

runner = AblationRunner(store)
result = await runner.compare(
    "thinking",
    "probe-42-a1b2c3d4", "baseline",
    "probe-42-9f8e7d6c", "ablated-thinking",
)

---

AnalysisReport

source

Aggregated error and score analysis for a completed run.

Attributes: total_records: Number of metric records analyzed. error_count: Number of error records analyzed. error_kinds: Error kind to occurrence count mapping. score_mean: Mean of records named score, or None. score_min: Minimum of records named score, or None. score_max: Maximum of records named score, or None. top_errors: Most frequent error records, most frequent first.

---

BatchEvaluationResult

source

Result of running evaluation on multiple samples.

Contains aggregated results and per-sample details.

---

Checkpoint

source

Content-addressed checkpoint of run state.

Attributes: run_id: Run the checkpoint belongs to. slug: Stable name of the checkpoint within the run. digest: SHA-256 digest of the canonicalized payload. payload: Checkpointed state. created_at: ISO-8601 creation timestamp.

---

ErrorAnalysis

source

Aggregate a run's tracked records into an analysis report.

Parameters

Parameter	Type	Description
`tracker`		Tracker holding the run's metric and error records.

__init__

def __init__(tracker: ExperimentTrackerProtocol) -> None

source

report

async def report(run_id: str) -> AnalysisReport

source

Produce an analysis report for a run.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.

Returns

Type	Description
AnalysisReport	Aggregated error kinds, score statistics, and top errors.

Raises

Exception	Description
AnalysisError	If the run is unknown to the tracker.

---

ErrorRecord

source

One error observed during an experiment run.

Attributes: kind: Error kind or code (e.g. LLM_RATE_LIMITED). message: Human-readable error message. step: Iteration or step index the error occurred at.

---

EvaluationConfig

source

Configuration for the evaluation subsystem.

Attributes: enabled: Enable the AI evaluation subsystem. default_threshold: Default score threshold for passing evaluations. embedding_model: Model to use for embedding-based evaluations. include_metadata: Whether to include metadata in run reports. max_samples: Maximum number of samples per evaluation run. max_retries: Maximum retries for failed evaluations. timeout_seconds: Timeout for evaluation execution in seconds.

Example

config = EvaluationConfig(
    default_threshold=0.9,
    embedding_model="text-embedding-3-large"
)

config = EvaluationConfig(
    default_threshold=0.9,
    embedding_model="text-embedding-3-large"
)

---

EvaluationDataset

source

A collection of evaluation samples.

Attributes: name: Name of the dataset. samples: List of evaluation samples. metadata: Additional dataset metadata.

---

EvaluationModule

source

Evaluation module for Oridecon applications.

Provides evaluator and harness support for AI model evaluation.

Usage

from oridecon.ai.evaluation import EvaluationModule
from oridecon.ai.evaluation.config import EvaluationConfig

@module(
    imports=[EvaluationModule.configure(EvaluationConfig(...))]
)
class AppModule(Module):
    pass

from oridecon.ai.evaluation import EvaluationModule
from oridecon.ai.evaluation.config import EvaluationConfig

@module(
    imports=[EvaluationModule.configure(EvaluationConfig(...))]
)
class AppModule(Module):
    pass

configure

def configure(
    cls,
    config: EvaluationConfig | None = None
) -> DynamicModule

source

Create an EvaluationModule with explicit configuration.

Parameters

Parameter	Type	Description
`config`	EvaluationConfig | None	EvaluationConfig or ``None`` for defaults.

Returns

Type	Description
DynamicModule	A DynamicModule descriptor.

stub

def stub(
    cls,
    config: EvaluationConfig | None = None
) -> DynamicModule

source

Create an EvaluationModule suitable for unit and integration testing.

Uses in-memory or no-op evaluator implementations with minimal side effects.

Parameters

Parameter	Type	Description
`config`	EvaluationConfig | None	Optional EvaluationConfig override. Uses safe test defaults when ``None``.

Returns

Type	Description
DynamicModule	A DynamicModule descriptor.

---

EvaluationProvider

source

Registers evaluation services with the DI container.

__init__

def __init__(config: EvaluationConfig | None = None) -> None

source

register

async def register(container: ContainerRegistrarProtocol) -> None

source

boot

async def boot(container: BootContainerProtocol) -> None

source

shutdown

async def shutdown() -> None

source

health_check

async def health_check(timeout: float = 5.0) -> HealthCheckResult

source

---

EvaluationResult

source

Result of an evaluation run on a single sample.

Attributes: score: The evaluation score (0.0 to 1.0). score_type: The type of scoring method used. feedback: Human-readable feedback about the evaluation. metrics: Additional metrics computed during evaluation.

---

EvaluationRunContext

source

Context for a single evaluation run.

Holds the dataset, evaluator, and configuration for an evaluation run.

---

EvaluationSample

source

A single sample in an evaluation dataset.

Attributes: id: Unique identifier for this sample. input: The input prompt or query. reference: The expected reference output. metadata: Additional metadata for this sample.

---

ExperimentConfig

source

Seed-and-config descriptor driving a reproducible experiment run.

Attributes: name: Experiment name; part of the derived run id. seed: Seed value; part of the derived run id. config: Knob configuration dict (canonical JSON order-insensitive). trials: Number of trials per run. Defaults to 1. metadata: Free-form run metadata.

---

ExperimentRun

source

A started experiment run with its identity and config digest.

Attributes: run_id: Stable identifier derived from name, seed, and config. experiment: Experiment name. seed: Seed value used for this run. config: Knob configuration dict. config_hash: SHA-256 digest of the canonicalized config. status: Lifecycle status of the run. started_at: ISO-8601 start timestamp. finished_at: ISO-8601 finish timestamp, or None while running.

---

FileCheckpointStore

source

Filesystem checkpoint store with digest verification.

Parameters

Parameter	Type	Description
`root`		Base directory for run artifacts. Defaults to ``runs``.

__init__

def __init__(root: str | Path = 'runs') -> None

source

save

async def save(
    run_id: str,
    slug: str,
    payload: dict[str, Any]
) -> Checkpoint

source

Persist a checkpoint for a run.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.
`slug`	str	Stable name of the checkpoint within the run.
`payload`	dict[str, Any]	State to checkpoint.

Returns

Type	Description
Checkpoint	The stored checkpoint with its content digest.

Raises

Exception	Description
CheckpointError	If the checkpoint cannot be written.

load

async def load(
    run_id: str,
    slug: str
) -> Checkpoint | None

source

Load a checkpoint, verifying its content digest.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.
`slug`	str	Checkpoint name.

Returns

Type	Description
Checkpoint | None	The verified checkpoint, or ``None`` when absent or tampered.

list

async def list(run_id: str) -> list[Checkpoint]

source

List all checkpoints for a run.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.

Returns

Type	Description
list[Checkpoint]	Checkpoints in creation order.

---

LocalTracker

source

JSON/JSONL-backed tracker persisting runs under ``/runs/``.

Parameters

Parameter	Type	Description
`root`		Base directory for run artifacts. Defaults to ``runs``.

__init__

def __init__(root: str | Path = 'runs') -> None

source

start

async def start(config: ExperimentConfig) -> ExperimentRun

source

Start (or resume) an experiment run for the given config.

Parameters

Parameter	Type	Description
`config`	ExperimentConfig	Seed and knob configuration of the run.

Returns

Type	Description
ExperimentRun	The started or resumed run.

Raises

Exception	Description
TrackingError	If the run manifest cannot be persisted.

log_metric

async def log_metric(
    run_id: str,
    name: str,
    value: float,
    step: int = 0
) -> None

source

Record a scalar metric for a run.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.
`name`	str	Metric name.
`value`	float	Metric value.
`step`	int	Step index the metric was recorded at. Defaults to 0.

Raises

Exception	Description
TrackingError	If the metric line cannot be appended.

log_error

async def log_error(
    run_id: str,
    kind: str,
    message: str,
    step: int = 0
) -> None

source

Record an error for a run.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.
`kind`	str	Error kind or code.
`message`	str	Human-readable error message.
`step`	int	Step index the error occurred at. Defaults to 0.

Raises

Exception	Description
TrackingError	If the error line cannot be appended.

metrics

async def metrics(run_id: str) -> list[MetricRecord]

source

Return all metric records for a run.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.

Returns

Type	Description
list[MetricRecord]	Metric records in recording order.

errors

async def errors(run_id: str) -> list[ErrorRecord]

source

Return all error records for a run.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.

Returns

Type	Description
list[ErrorRecord]	Error records in recording order.

snapshot

async def snapshot(run_id: str) -> dict[str, Any]

source

Return a compact summary of a run’s current state.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.

Returns

Type	Description
dict[str, Any]	Latest value per metric, error counts, and run metadata.

Raises

Exception	Description
TrackingError	If the run is unknown.

resume

async def resume(run_id: str) -> ExperimentRun | None

source

Return an already-started run, or None when unknown.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.

Returns

Type	Description
ExperimentRun | None	The existing run, or ``None``.

finish

async def finish(
    run_id: str,
    status: RunStatus = RunStatus.COMPLETED
) -> None

source

Mark a run finished.

Parameters

Parameter	Type	Description
`run_id`	str	Run identifier.
`status`	RunStatus	Terminal status. Defaults to ``COMPLETED``.

Raises

Exception	Description
TrackingError	If the run is unknown.

---

MetricRecord

source

A single scalar metric recorded during an experiment run.

Attributes: step: Iteration or step index the metric was recorded at. name: Metric name (e.g. tokens, latency_ms, score). value: Numeric metric value.

---

RunReport

source

Report from running an evaluator on a dataset.

Attributes: dataset_name: Name of the evaluated dataset. evaluator_name: Name of the evaluator used. total_samples: Total number of samples evaluated. passed_samples: Number of samples that passed the evaluation. average_score: Average score across all samples. results: Individual sample results. metadata: Additional report metadata.

---

RunStatus

source

Lifecycle status of an experiment run.

---

Functions

make_run_id

make_run_id

def make_run_id(
    name: str,
    seed: int,
    config: dict[str, Any]
) -> str

source

Derive a deterministic run id from name, seed, and canonical config.

Parameters

Parameter	Type	Description
`name`	str	Experiment name.
`seed`	int	Seed value.
`config`	dict[str, Any]	Knob configuration dict.

Returns

Type	Description
str	Run id of the form ``--<8-char digest>``.

Example

run_id = make_run_id("probe", 42, {"model": "gpt-4o"})
assert run_id == make_run_id("probe", 42, {"model": "gpt-4o"})

---

Exceptions

AblationError

source

Raised when an ablation references unknown checkpoints.

---

AnalysisError

source

Raised when a run analysis cannot be produced.

---

CheckpointError

source

Raised when a checkpoint is missing or fails digest verification.

---

DatasetError

source

Raised when there's an error with the evaluation dataset.

---

EvaluationConfigError

source

Raised when evaluation configuration is invalid.

---

EvaluatorNotFoundError

source

Raised when a requested evaluator cannot be found.

---

HarnessError

source

Raised when the evaluation harness encounters an error.

---

TrackingError

source

Raised when experiment tracking cannot persist or read run state.

---