harnest.evaluation

All Python modules · Import and compatibility guidance

Metric return values

Return value	Scoring policy
`MetricScore(0.9, "Evidence")`	Scores the conversation once, on its final actual turn; earlier turns remain unscored
A list or tuple of `MetricScore` values	Requires one value per actual turn; the overall score is the mean of scored turns
`MetricScore(None, "Evidence unavailable")`	Marks the score `NOT_EVALUATED`; an entirely unscored metric cannot pass
Native ADK `EvaluationResult`	Passes through unchanged for advanced result control

Native scorer contract

from google.adk.evaluation.conversation_scenarios import ConversationScenario
from google.adk.evaluation.eval_case import Invocation
from google.adk.evaluation.eval_metrics import EvalMetric
from google.adk.evaluation.evaluator import EvaluationResult

def required_location_terms(
    metric: EvalMetric,
    actual: list[Invocation],
    expected: list[Invocation] | None,
    scenario: ConversationScenario | None,
) -> EvaluationResult:
    """Signature only; see Create custom evals for the complete implementation."""
    ...

The evaluator calls the function with four positional values:

Argument	Value
`metric`	Metric name, criterion, and registered function path; read the threshold from `metric.criterion.threshold`, not the deprecated `metric.threshold` field
`actual`	Ordered invocations captured from one case, not the whole eval set
`expected`	Golden invocations for a static case, otherwise `None`
`scenario`	ADK `ConversationScenario` for a simulated case, otherwise `None`

Scorers may be sync or async; use async I/O for network calls. They receive ADK evaluation inputs, not a Harnest runtime or storage client. Actual and golden invocation counts can differ. Check the index before accessing expected[index], and handle expected=None for simulations.

Native result contract

A scored EvaluationResult requires one ordered PerInvocationResult per actual invocation. Entirely NOT_EVALUATED results are exempt.

Result field	Contract
`overall_score`	The aggregate score, or `None` when not evaluated
`overall_eval_status`	An explicit ADK `EvalStatus` consistent with the score and threshold
`per_invocation_results`	One row per actual turn for a scored metric
Each row’s `actual_invocation`	The corresponding captured invocation
Each row’s `expected_invocation`	The matching golden invocation when available, otherwise `None`
Each row’s `score` and `eval_status`	The turn’s numeric score and explicit status, or `None` and `NOT_EVALUATED` when unscored

Keep aggregate and per-turn scores consistent: the CLI gate compares the mean of scored turns with the threshold. For final-turn scoring, mark earlier rows NOT_EVALUATED and use the last turn’s score as the overall score. Entirely unscored metrics cannot pass. See the walkthrough for mean-of-turns scoring.

EvalRunResult envelope

The current contract is identified by:

{
  "apiVersion": "harnest.dev/v1alpha1",
  "kind": "EvalRunResult"
}

Branch consumers on apiVersion. The result schema is versioned independently from the installed ADK’s nested case models.

{
  "apiVersion": "harnest.dev/v1alpha1",
  "kind": "EvalRunResult",
  "createdAt": "2026-09-03T10:30:00+00:00",
  "framework": "adk",
  "trajectory": "business",
  "status": "failed",
  "summary": {
    "suiteCount": 1,
    "caseCount": 1,
    "passedCases": 0,
    "failedCases": 1,
    "notEvaluatedCases": 0
  },
  "evalSetResults": [
    {
      "appName": "harnest_eval",
      "evalSetId": "city-facts",
      "status": "failed",
      "evalCaseResults": [
        {
          "evalSetId": "city-facts",
          "evalId": "verify_paris",
          "finalEvalStatus": "failed",
          "overallEvalMetricResults": [
            {
              "metricName": "response_match_score",
              "threshold": 0.8,
              "score": 0.6,
              "evalStatus": "failed",
              "details": {"rubricScores": null}
            }
          ],
          "evalMetricResultPerInvocation": [
            {
              "actualInvocation": {},
              "expectedInvocation": {},
              "evalMetricResults": []
            }
          ],
          "sessionId": "___eval___session___...",
          "sessionDetails": {},
          "userId": "eval-user"
        }
      ]
    }
  ]
}

The abbreviated empty invocation and session objects above stand in for complete ADK-shaped data in a real result.

Top-level fields

Field	Meaning
`createdAt`	UTC creation time
`framework`	Agent runtime: `adk` or `langgraph`. Both use ADK’s evaluation contract.
`trajectory`	Effective `business` or `strict` policy
`status`	`passed`, `failed`, `not_evaluated`, or `error`
`summary`	Suite and case counts by status
`evalSetResults`	Ordered suite results and their complete case results
`error`	Infrastructure exception type and a fixed privacy-safe message; present when `status` is `error`

Case diagnostics

Each evalCaseResults item preserves:

Overall result, score, criterion, and details for every metric.
Per-invocation metric results.
Complete actual invocations and optional expected invocations.
User content, final responses, intermediate events, tool evidence, and applicable rubrics carried by those invocations.
Evaluator session ID, user ID, and session details, including state and recorded events.
Rubric scores and rationales in each metric’s details.rubricScores when the evaluator supplies them.

Statuses are lowercase names. not_evaluated means no score; the CLI averages scored turns only.

Infrastructure errors

If evaluation has started and a provider, adapter, or evaluator raises an infrastructure error, Harnest emits or writes the partial result before reporting the original command failure. Its top-level error excerpt looks like this:

{
  "status": "error",
  "error": {
    "type": "RuntimeError",
    "message": "evaluation infrastructure failed"
  },
  "evalSetResults": []
}

Harnest omits the original exception message, traceback, and internals from this field because provider errors can contain credentials or request content. Failures before the eval lane starts, such as compilation, unit tests, or invalid configuration, may produce no result file. EvaluationExecutionError means execution or scoring failed without metric results. Follow its safe diagnostic message and inspect the local traceback. Fix runtime problems before reassessing quality.

Judge configuration

Property	Default	Applies to	Purpose
`threshold`	Required	Every metric	Pass boundary
`judgeModelOptions.judgeModel`	Configured `OPENAI_MODEL`	Model-judged metrics	Omit to use the shared compatible API; explicit IDs select an ADK model provider
`judgeModelOptions.judgeModelConfig`	None	Model-judged metrics	Google GenAI generation options for judge calls
`judgeModelOptions.numSamples`	`5`	Final-response match, rubric metrics, and simulator-quality metrics	Judge calls per invocation before aggregation
`judgeModelOptions.parallelismLimit`	`1`	Final-response match and rubric metrics	Maximum concurrent judge calls
`includeIntermediateResponsesInFinal`	`false`	`final_response_match_v2` and rubric-based final response quality	Include visible text emitted before tool calls with the final output

Exports

Name	Kind
`EVAL_TRAJECTORIES`	Value or type alias
`EvalRunnerContext`	Class
`EvalRunnerResult`	Class
`EvaluationError`	Class
`MetricContext`	Class
`MetricScore`	Class
`adk_eval_agent_module`	Function
`eval_config`	Function
`eval_dependencies`	Function
`metric`	Function
`require_eval_trajectory`	Function
`supported_metric_names`	Function

EVAL_TRAJECTORIES

EVAL_TRAJECTORIES = frozenset(['business', 'strict'])

Example

from harnest.evaluation import EVAL_TRAJECTORIES

EvalRunnerContext

One runner’s options, agent HTTP client, project inputs, and private report directory.

EvalRunnerContext(
    client: httpx.Client,
    project_dir: Path,
    report_dir: Path,
    options: Mapping[str, Any],
) -> None

Fields (— means no declared default):

Field	Type	Default
`client`	`httpx.Client`	`—`
`project_dir`	`Path`	`—`
`report_dir`	`Path`	`—`
`options`	`Mapping[str, Any]`	`—`

Parameters and example

Parameter	Type	Default
`client`	`httpx.Client`	Required
`project_dir`	`Path`	Required
`report_dir`	`Path`	Required
`options`	`Mapping[str, Any]`	Required

from harnest.evaluation import EvalRunnerContext

context = EvalRunnerContext(client=..., project_dir=..., report_dir=..., options=...)

EvalRunnerResult

A quality verdict and optional report files relative to the runner’s directory.

EvalRunnerResult(
    status: Literal['passed', 'failed', 'not_evaluated'],
    artifacts: tuple[str, ...] = (),
) -> None

Fields (— means no declared default):

Field	Type	Default
`status`	`Literal['passed', 'failed', 'not_evaluated']`	`—`
`artifacts`	`tuple[str, ...]`	`()`

Parameters and example

Parameter	Type	Default
`status`	`Literal['passed', 'failed', 'not_evaluated']`	Required
`artifacts`	`tuple[str, ...]`	`()`

from harnest.evaluation import EvalRunnerResult

result = EvalRunnerResult(status=...)

EvaluationError

An evaluation request cannot be executed with the installed runtime.

Example

from harnest.evaluation import EvaluationError

try:
    ...
except EvaluationError as error:
    ...

MetricContext

A case’s native ADK evidence, available to sync and async Harnest scorers.

MetricContext(
    metric: EvalMetric,
    actual: tuple[Invocation, ...],
    expected: tuple[Invocation, ...] | None = None,
    scenario: ConversationScenario | None = None,
) -> None

Fields (— means no declared default):

Field	Type	Default
`metric`	`EvalMetric`	`—`
`actual`	`tuple[Invocation, ...]`	`—`
`expected`	`tuple[Invocation, ...] \| None`	`None`
`scenario`	`ConversationScenario \| None`	`None`

Parameters and example

Parameter	Type	Default
`metric`	`EvalMetric`	Required
`actual`	`tuple[Invocation, ...]`	Required
`expected`	`tuple[Invocation, ...] \| None`	`None`
`scenario`	`ConversationScenario \| None`	`None`

from harnest.evaluation import MetricContext

context = MetricContext(metric=..., actual=...)

MetricContext.threshold

Read the authored criterion instead of ADK’s deprecated threshold field.

Example

from harnest.evaluation import MetricContext

context: MetricContext
value = context.threshold

MetricScore

One finite score and optional human-readable evidence; None means unscored.

MetricScore(score: float | None, evidence: str | None = None) -> None

Fields (— means no declared default):

Field	Type	Default
`score`	`float \| None`	`—`
`evidence`	`str \| None`	`None`

Parameters and example

Parameter	Type	Default
`score`	`float \| None`	Required
`evidence`	`str \| None`	`None`

from harnest.evaluation import MetricScore

score = MetricScore(score=...)

adk_eval_agent_module

Expose isolated native plugins with managed identity and error reporting.

adk_eval_agent_module(application: Any) -> Iterator[str]

Parameters and example

Parameter	Type	Default
`application`	`Any`	Required

from harnest.evaluation import adk_eval_agent_module

with adk_eval_agent_module(application=...):
    ...

eval_config

Load criteria with shared model defaults and the selected trajectory.

eval_config(suite: EvalSuite, trajectory: str) -> Any

Parameters and example

Parameter	Type	Default
`suite`	`EvalSuite`	Required
`trajectory`	`str`	Required

from harnest.evaluation import eval_config

result = eval_config(suite=..., trajectory=...)

eval_dependencies

Validate the shared evaluation engine before loading either framework adapter.

eval_dependencies() -> tuple[Any, Any]

Example

from harnest.evaluation import eval_dependencies

result = eval_dependencies()

metric

Adapt a context-based scorer to ADK’s four-argument custom metric contract. Return a MetricScore for the whole conversation, a sequence with one score per actual invocation, or a native ADK EvaluationResult. Native ADK scorers can also remain undecorated and register through the same customMetrics map.

metric(function: Callable[[MetricContext], Any]) -> Callable[..., Any]

Parameters and example

Parameter	Type	Default
`function`	`Callable[[MetricContext], Any]`	Required

from harnest.evaluation import metric

@metric
def my_function(context):
    ...

require_eval_trajectory

Reject unknown tool-trajectory policies at every evaluation entrypoint.

require_eval_trajectory(trajectory: str) -> None

Parameters and example

Parameter	Type	Default
`trajectory`	`str`	Required

from harnest.evaluation import require_eval_trajectory

require_eval_trajectory(trajectory=...)

supported_metric_names

Reflect the installed ADK metric registry instead of freezing an allowlist.

supported_metric_names() -> tuple[str, ...]

Example

from harnest.evaluation import supported_metric_names

result = supported_metric_names()

Overview

Get started

Harnest Studio

Guides

API reference

harnest.evaluation

Metric return values

Native scorer contract

Native result contract

EvalRunResult envelope

Top-level fields

Case diagnostics

Infrastructure errors

Judge configuration

Exports

EVAL_TRAJECTORIES

EvalRunnerContext

EvalRunnerResult

EvaluationError

MetricContext

MetricContext.threshold

MetricScore

adk_eval_agent_module

eval_config

eval_dependencies

metric

require_eval_trajectory

supported_metric_names

​Metric return values

​Native scorer contract

​Native result contract

​EvalRunResult envelope

​Top-level fields

​Case diagnostics

​Infrastructure errors

​Judge configuration

​Exports

​EVAL_TRAJECTORIES

​EvalRunnerContext

​EvalRunnerResult

​EvaluationError

​MetricContext

​MetricContext.threshold

​MetricScore

​adk_eval_agent_module

​eval_config

​eval_dependencies

​metric

​require_eval_trajectory

​supported_metric_names

Metric return values

Native scorer contract

Native result contract

EvalRunResult envelope

Top-level fields

Case diagnostics

Infrastructure errors

Judge configuration

Exports

EVAL_TRAJECTORIES

EvalRunnerContext

EvalRunnerResult

EvaluationError

MetricContext

MetricContext.threshold

MetricScore

adk_eval_agent_module

eval_config

eval_dependencies

metric

require_eval_trajectory

supported_metric_names