> ## Documentation Index
> Fetch the complete documentation index at: https://usefused.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# harnest.evaluation

> Public Python signatures, fields, and methods.

[All Python modules](https://usefused.com/docs/harnest/reference/api/overview.md) · [Import and compatibility guidance](https://usefused.com/docs/harnest/build/python-imports.md)

## Metric return values

| Return value | Scoring policy |
| - | - |
| `MetricScore(0.9, "Evidence")` | Scores the conversation once, on its final actual turn; earlier turns remain unscored |
| A list or tuple of `MetricScore` values | Requires one value per actual turn; the overall score is the mean of scored turns |
| `MetricScore(None, "Evidence unavailable")` | Marks the score `NOT_EVALUATED`; an entirely unscored metric cannot pass |
| Native ADK `EvaluationResult` | Passes through unchanged for advanced result control |

## Native scorer contract

```python theme={null}
from google.adk.evaluation.conversation_scenarios import ConversationScenario
from google.adk.evaluation.eval_case import Invocation
from google.adk.evaluation.eval_metrics import EvalMetric
from google.adk.evaluation.evaluator import EvaluationResult

def required_location_terms(
    metric: EvalMetric,
    actual: list[Invocation],
    expected: list[Invocation] | None,
    scenario: ConversationScenario | None,
) -> EvaluationResult:
    """Signature only; see Create custom evals for the complete implementation."""
    ...
```

The evaluator calls the function with four positional values:

| Argument | Value |
| - | - |
| `metric` | Metric name, criterion, and registered function path; read the threshold from `metric.criterion.threshold`, not the deprecated `metric.threshold` field |
| `actual` | Ordered invocations captured from one case, not the whole eval set |
| `expected` | Golden invocations for a static case, otherwise `None` |
| `scenario` | ADK `ConversationScenario` for a simulated case, otherwise `None` |

Scorers may be sync or async; use async I/O for network calls. They receive ADK evaluation inputs, not a Harnest runtime or storage client.

Actual and golden invocation counts can differ. Check the index before accessing `expected[index]`, and handle `expected=None` for simulations.

## Native result contract

A scored `EvaluationResult` requires one ordered `PerInvocationResult` per actual invocation. Entirely `NOT_EVALUATED` results are exempt.

| Result field | Contract |
| - | - |
| `overall_score` | The aggregate score, or `None` when not evaluated |
| `overall_eval_status` | An explicit ADK `EvalStatus` consistent with the score and threshold |
| `per_invocation_results` | One row per actual turn for a scored metric |
| Each row's `actual_invocation` | The corresponding captured invocation |
| Each row's `expected_invocation` | The matching golden invocation when available, otherwise `None` |
| Each row's `score` and `eval_status` | The turn's numeric score and explicit status, or `None` and `NOT_EVALUATED` when unscored |

Keep aggregate and per-turn scores consistent: the CLI gate compares the mean of scored turns with the threshold.

For final-turn scoring, mark earlier rows `NOT_EVALUATED` and use the last turn’s score as the overall score. Entirely unscored metrics cannot pass. See the [walkthrough](https://usefused.com/docs/harnest/build/evaluations/create-custom-evals.md) for mean-of-turns scoring.

## EvalRunResult envelope

The current contract is identified by:

```json theme={null}
{
  "apiVersion": "harnest.dev/v1alpha1",
  "kind": "EvalRunResult"
}
```

Branch consumers on `apiVersion`. The result schema is versioned independently from the installed ADK's nested case models.

```json theme={null}
{
  "apiVersion": "harnest.dev/v1alpha1",
  "kind": "EvalRunResult",
  "createdAt": "2026-09-03T10:30:00+00:00",
  "framework": "adk",
  "trajectory": "business",
  "status": "failed",
  "summary": {
    "suiteCount": 1,
    "caseCount": 1,
    "passedCases": 0,
    "failedCases": 1,
    "notEvaluatedCases": 0
  },
  "evalSetResults": [
    {
      "appName": "harnest_eval",
      "evalSetId": "city-facts",
      "status": "failed",
      "evalCaseResults": [
        {
          "evalSetId": "city-facts",
          "evalId": "verify_paris",
          "finalEvalStatus": "failed",
          "overallEvalMetricResults": [
            {
              "metricName": "response_match_score",
              "threshold": 0.8,
              "score": 0.6,
              "evalStatus": "failed",
              "details": {"rubricScores": null}
            }
          ],
          "evalMetricResultPerInvocation": [
            {
              "actualInvocation": {},
              "expectedInvocation": {},
              "evalMetricResults": []
            }
          ],
          "sessionId": "___eval___session___...",
          "sessionDetails": {},
          "userId": "eval-user"
        }
      ]
    }
  ]
}
```

The abbreviated empty invocation and session objects above stand in for complete ADK-shaped data in a real result.

### Top-level fields

| Field | Meaning |
| - | - |
| `createdAt` | UTC creation time |
| `framework` | Agent runtime: `adk` or `langgraph`. Both use ADK's evaluation contract. |
| `trajectory` | Effective `business` or `strict` policy |
| `status` | `passed`, `failed`, `not_evaluated`, or `error` |
| `summary` | Suite and case counts by status |
| `evalSetResults` | Ordered suite results and their complete case results |
| `error` | Infrastructure exception type and a fixed privacy-safe message; present when `status` is `error` |

### Case diagnostics

Each `evalCaseResults` item preserves:

* Overall result, score, criterion, and details for every metric.
* Per-invocation metric results.
* Complete actual invocations and optional expected invocations.
* User content, final responses, intermediate events, tool evidence, and applicable rubrics carried by those invocations.
* Evaluator session ID, user ID, and session details, including state and recorded events.
* Rubric scores and rationales in each metric's `details.rubricScores` when the evaluator supplies them.

Statuses are lowercase names. `not_evaluated` means no score; the CLI averages scored turns only.

### Infrastructure errors

If evaluation has started and a provider, adapter, or evaluator raises an infrastructure error, Harnest emits or writes the partial result before reporting the original command failure. Its top-level error excerpt looks like this:

```json theme={null}
{
  "status": "error",
  "error": {
    "type": "RuntimeError",
    "message": "evaluation infrastructure failed"
  },
  "evalSetResults": []
}
```

Harnest omits the original exception message, traceback, and internals from this field because provider errors can contain credentials or request content. Failures before the eval lane starts, such as compilation, unit tests, or invalid configuration, may produce no result file.

`EvaluationExecutionError` means execution or scoring failed without metric results. Follow its safe diagnostic message and inspect the local traceback. Fix runtime problems before reassessing quality.

## Judge configuration

| Property | Default | Applies to | Purpose |
| - | -: | - | - |
| `threshold` | Required | Every metric | Pass boundary |
| `judgeModelOptions.judgeModel` | Configured `OPENAI_MODEL` | Model-judged metrics | Omit to use the shared compatible API; explicit IDs select an ADK model provider |
| `judgeModelOptions.judgeModelConfig` | None | Model-judged metrics | Google GenAI generation options for judge calls |
| `judgeModelOptions.numSamples` | `5` | Final-response match, rubric metrics, and simulator-quality metrics | Judge calls per invocation before aggregation |
| `judgeModelOptions.parallelismLimit` | `1` | Final-response match and rubric metrics | Maximum concurrent judge calls |
| `includeIntermediateResponsesInFinal` | `false` | `final_response_match_v2` and rubric-based final response quality | Include visible text emitted before tool calls with the final output |

## Exports

| Name | Kind |
| - | - |
| [`EVAL_TRAJECTORIES`](#eval_trajectories) | Value or type alias |
| [`EvalRunnerContext`](#evalrunnercontext) | Class |
| [`EvalRunnerResult`](#evalrunnerresult) | Class |
| [`EvaluationError`](#evaluationerror) | Class |
| [`MetricContext`](#metriccontext) | Class |
| [`MetricScore`](#metricscore) | Class |
| [`adk_eval_agent_module`](#adk_eval_agent_module) | Function |
| [`eval_config`](#eval_config) | Function |
| [`eval_dependencies`](#eval_dependencies) | Function |
| [`metric`](#metric) | Function |
| [`require_eval_trajectory`](#require_eval_trajectory) | Function |
| [`supported_metric_names`](#supported_metric_names) | Function |

## EVAL\_TRAJECTORIES

```python theme={null}
EVAL_TRAJECTORIES = frozenset(['business', 'strict'])
```

> [accordion] Example
>
> ```python theme={null}
> from harnest.evaluation import EVAL_TRAJECTORIES
> ```

## EvalRunnerContext

One runner's options, agent HTTP client, project inputs, and private report directory.

```python theme={null}
EvalRunnerContext(
    client: httpx.Client,
    project_dir: Path,
    report_dir: Path,
    options: Mapping[str, Any],
) -> None
```

Fields (`—` means no declared default):

| Field | Type | Default |
| - | - | - |
| `client` | `httpx.Client` | `—` |
| `project_dir` | `Path` | `—` |
| `report_dir` | `Path` | `—` |
| `options` | `Mapping[str, Any]` | `—` |

> [accordion] Parameters and example
>
> | Parameter | Type | Default |
> | - | - | - |
> | `client` | `httpx.Client` | Required |
> | `project_dir` | `Path` | Required |
> | `report_dir` | `Path` | Required |
> | `options` | `Mapping[str, Any]` | Required |
>
> ```python theme={null}
> from harnest.evaluation import EvalRunnerContext
>
> context = EvalRunnerContext(client=..., project_dir=..., report_dir=..., options=...)
> ```

## EvalRunnerResult

A quality verdict and optional report files relative to the runner's directory.

```python theme={null}
EvalRunnerResult(
    status: Literal['passed', 'failed', 'not_evaluated'],
    artifacts: tuple[str, ...] = (),
) -> None
```

Fields (`—` means no declared default):

| Field | Type | Default |
| - | - | - |
| `status` | `Literal['passed', 'failed', 'not_evaluated']` | `—` |
| `artifacts` | `tuple[str, ...]` | `()` |

> [accordion] Parameters and example
>
> | Parameter | Type | Default |
> | - | - | - |
> | `status` | `Literal['passed', 'failed', 'not_evaluated']` | Required |
> | `artifacts` | `tuple[str, ...]` | `()` |
>
> ```python theme={null}
> from harnest.evaluation import EvalRunnerResult
>
> result = EvalRunnerResult(status=...)
> ```

## EvaluationError

An evaluation request cannot be executed with the installed runtime.

> [accordion] Example
>
> ```python theme={null}
> from harnest.evaluation import EvaluationError
>
> try:
>     ...
> except EvaluationError as error:
>     ...
> ```

## MetricContext

A case's native ADK evidence, available to sync and async Harnest scorers.

```python theme={null}
MetricContext(
    metric: EvalMetric,
    actual: tuple[Invocation, ...],
    expected: tuple[Invocation, ...] | None = None,
    scenario: ConversationScenario | None = None,
) -> None
```

Fields (`—` means no declared default):

| Field | Type | Default |
| - | - | - |
| `metric` | `EvalMetric` | `—` |
| `actual` | `tuple[Invocation, ...]` | `—` |
| `expected` | `tuple[Invocation, ...] \| None` | `None` |
| `scenario` | `ConversationScenario \| None` | `None` |

> [accordion] Parameters and example
>
> | Parameter | Type | Default |
> | - | - | - |
> | `metric` | `EvalMetric` | Required |
> | `actual` | `tuple[Invocation, ...]` | Required |
> | `expected` | `tuple[Invocation, ...] \| None` | `None` |
> | `scenario` | `ConversationScenario \| None` | `None` |
>
> ```python theme={null}
> from harnest.evaluation import MetricContext
>
> context = MetricContext(metric=..., actual=...)
> ```

### MetricContext.threshold

Read the authored criterion instead of ADK's deprecated threshold field.

> [accordion] Example
>
> ```python theme={null}
> from harnest.evaluation import MetricContext
>
> context: MetricContext
> value = context.threshold
> ```

## MetricScore

One finite score and optional human-readable evidence; None means unscored.

```python theme={null}
MetricScore(score: float | None, evidence: str | None = None) -> None
```

Fields (`—` means no declared default):

| Field | Type | Default |
| - | - | - |
| `score` | `float \| None` | `—` |
| `evidence` | `str \| None` | `None` |

> [accordion] Parameters and example
>
> | Parameter | Type | Default |
> | - | - | - |
> | `score` | `float \| None` | Required |
> | `evidence` | `str \| None` | `None` |
>
> ```python theme={null}
> from harnest.evaluation import MetricScore
>
> score = MetricScore(score=...)
> ```

## adk\_eval\_agent\_module

Expose isolated native plugins with managed identity and error reporting.

```python theme={null}
adk_eval_agent_module(application: Any) -> Iterator[str]
```

> [accordion] Parameters and example
>
> | Parameter | Type | Default |
> | - | - | - |
> | `application` | `Any` | Required |
>
> ```python theme={null}
> from harnest.evaluation import adk_eval_agent_module
>
> with adk_eval_agent_module(application=...):
>     ...
> ```

## eval\_config

Load criteria with shared model defaults and the selected trajectory.

```python theme={null}
eval_config(suite: EvalSuite, trajectory: str) -> Any
```

> [accordion] Parameters and example
>
> | Parameter | Type | Default |
> | - | - | - |
> | `suite` | `EvalSuite` | Required |
> | `trajectory` | `str` | Required |
>
> ```python theme={null}
> from harnest.evaluation import eval_config
>
> result = eval_config(suite=..., trajectory=...)
> ```

## eval\_dependencies

Validate the shared evaluation engine before loading either framework adapter.

```python theme={null}
eval_dependencies() -> tuple[Any, Any]
```

> [accordion] Example
>
> ```python theme={null}
> from harnest.evaluation import eval_dependencies
>
> result = eval_dependencies()
> ```

## metric

Adapt a context-based scorer to ADK's four-argument custom metric contract.

Return a MetricScore for the whole conversation, a sequence with one score
per actual invocation, or a native ADK EvaluationResult. Native ADK scorers
can also remain undecorated and register through the same customMetrics map.

```python theme={null}
metric(function: Callable[[MetricContext], Any]) -> Callable[..., Any]
```

> [accordion] Parameters and example
>
> | Parameter | Type | Default |
> | - | - | - |
> | `function` | `Callable[[MetricContext], Any]` | Required |
>
> ```python theme={null}
> from harnest.evaluation import metric
>
> @metric
> def my_function(context):
>     ...
> ```

## require\_eval\_trajectory

Reject unknown tool-trajectory policies at every evaluation entrypoint.

```python theme={null}
require_eval_trajectory(trajectory: str) -> None
```

> [accordion] Parameters and example
>
> | Parameter | Type | Default |
> | - | - | - |
> | `trajectory` | `str` | Required |
>
> ```python theme={null}
> from harnest.evaluation import require_eval_trajectory
>
> require_eval_trajectory(trajectory=...)
> ```

## supported\_metric\_names

Reflect the installed ADK metric registry instead of freezing an allowlist.

```python theme={null}
supported_metric_names() -> tuple[str, ...]
```

> [accordion] Example
>
> ```python theme={null}
> from harnest.evaluation import supported_metric_names
>
> result = supported_metric_names()
> ```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.