> ## Documentation Index
> Fetch the complete documentation index at: https://usefused.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Run and inspect evaluation results
> Run every eval suite, save the versioned EvalRunResult JSON, and keep sensitive diagnostics safe in CI.
`harnest test --evals` compiles the agent, runs the selected Python test lanes, evaluates every valid root eval set, and returns a nonzero status when the quality gate fails.
## Run evaluations
```bash theme={null}
harnest test AGENT_DIR --evals
```
The unit lane runs first. Add `--smoke` to run smoke tests before evaluation:
```bash theme={null}
harnest test AGENT_DIR --smoke --evals
```
If a Python test lane fails, later lanes do not start. During evaluation, a scored failure in one eval set does not prevent independent later sets from running. The final result therefore covers every suite reached by the eval lane.
Harnest runs each suite once and processes suite files in deterministic filename order. Judge metrics can still make multiple calls according to their `numSamples` setting.
Native ADK evaluations run with Harnest's managed invocation context: session identity, built-in Agent Skills tools such as `list_skills`, explicitly assigned sandboxes, and configured context resources and credentials are available. ADK still owns the evaluation runner; this does not promise identical lifecycle or plugin behavior to normal serving. Context is available only while that invocation is active; do not retain it for background work after the invocation finishes.
Evaluation session data stays in memory, separate from your deployed session backend. The CLI starts and closes its evaluation resources; the playground reuses the running application's resources. The isolated CLI ADK evaluation runtime does not start task workers or cron schedules, so it does not consume unrelated production queue rows; test queued workflows with an appropriate smoke or integration runtime instead. This session isolation does not make evaluation a dry run: tools, sandboxes, and external services can still have real side effects.
For native ADK workflow roots that use its legacy runner, such as `SequentialAgent` or a non-`LlmAgent` `BaseAgent`, Harnest rejects a plugin's `before_run_callback` that returns replacement `Content`. Interrupting that response stream can skip invocation cleanup. Return `None` from the callback and produce the response from the agent or a tool instead. This restriction does not apply to the normal `LlmAgent`/node execution path.
## Select the trajectory policy
```bash theme={null}
# Required business calls in order; extra helper calls are allowed.
harnest test AGENT_DIR --evals --eval-trajectory business
# Tool names, order, arguments, and call count must match exactly.
harnest test AGENT_DIR --evals --eval-trajectory strict
```
`business` is the default. The flag affects only `tool_trajectory_avg_score`.
## Save the complete result
Without `--no-output`, Harnest prints detailed evaluator output and a complete structured JSON `EvalRunResult` after evaluation. Select a machine-readable file instead of parsing mixed terminal output:
```bash theme={null}
harnest test AGENT_DIR --evals \
--eval-output artifacts/eval-result.json
```
Harnest creates missing parent directories and atomically replaces the selected file. It does not append to an existing file or create an implicit result history.
For quiet CI output, combine the explicit file with `--no-output`:
```bash theme={null}
harnest test AGENT_DIR --evals --no-output \
--eval-output artifacts/eval-result.json
```
`--no-output` suppresses unit, smoke, evaluator, and final result output in the terminal. It does not suppress an explicitly selected result file. `--eval-output` requires `--evals`.
## Understand `EvalRunResult`
The current contract is identified by:
```json theme={null}
{
"apiVersion": "harnest.dev/v1alpha1",
"kind": "EvalRunResult"
}
```
Branch consumers on `apiVersion`. The result schema is versioned independently from the installed ADK's nested case models.
```json theme={null}
{
"apiVersion": "harnest.dev/v1alpha1",
"kind": "EvalRunResult",
"createdAt": "2026-09-03T10:30:00+00:00",
"framework": "adk",
"trajectory": "business",
"status": "failed",
"summary": {
"suiteCount": 1,
"caseCount": 1,
"passedCases": 0,
"failedCases": 1,
"notEvaluatedCases": 0
},
"evalSetResults": [
{
"appName": "harnest_eval",
"evalSetId": "city-facts",
"status": "failed",
"evalCaseResults": [
{
"evalSetId": "city-facts",
"evalId": "verify_paris",
"finalEvalStatus": "failed",
"overallEvalMetricResults": [
{
"metricName": "response_match_score",
"threshold": 0.8,
"score": 0.6,
"evalStatus": "failed",
"details": {"rubricScores": null}
}
],
"evalMetricResultPerInvocation": [
{
"actualInvocation": {},
"expectedInvocation": {},
"evalMetricResults": []
}
],
"sessionId": "___eval___session___...",
"sessionDetails": {},
"userId": "eval-user"
}
]
}
]
}
```
The abbreviated empty invocation and session objects above stand in for complete ADK-shaped data in a real result.
### Top-level fields
| Field | Meaning |
| ---------------- | ------------------------------------------------------------------------------------------------ |
| `createdAt` | UTC creation time |
| `framework` | Agent runtime: `adk` or `langgraph`. Both use ADK's evaluation contract. |
| `trajectory` | Effective `business` or `strict` policy |
| `status` | `passed`, `failed`, `not_evaluated`, or `error` |
| `summary` | Suite and case counts by status |
| `evalSetResults` | Ordered suite results and their complete case results |
| `error` | Infrastructure exception type and a fixed privacy-safe message; present when `status` is `error` |
### Case diagnostics
Each `evalCaseResults` item preserves:
* Overall result, score, criterion, and details for every metric.
* Per-invocation metric results.
* Complete actual invocations and optional expected invocations.
* User content, final responses, intermediate events, tool evidence, and applicable rubrics carried by those invocations.
* Evaluator session ID, user ID, and session details, including state and recorded events.
* Rubric scores and rationales in each metric's `details.rubricScores` when the evaluator supplies them.
Statuses are lowercase names rather than ADK enum ordinals. `not_evaluated` means the row has no score. Earlier turns of a final-turn-only metric can have this status without failing the CLI gate: ADK averages the available scored turns. A metric with no scored turns does not pass. Inspect the overall result together with the scored turn evidence.
### Infrastructure errors
If evaluation has started and a provider, adapter, or evaluator raises an infrastructure error, Harnest emits or writes the partial result before reporting the original command failure. Its top-level error excerpt looks like this:
```json theme={null}
{
"status": "error",
"error": {
"type": "RuntimeError",
"message": "evaluation infrastructure failed"
},
"evalSetResults": []
}
```
Harnest omits the original exception message, traceback, and internals from this field because provider errors can contain credentials or request content. Failures before the eval lane starts, such as compilation, unit tests, or invalid configuration, may produce no result file.
If native ADK cannot complete execution or scoring and produces no metric results, Harnest reports `EvaluationExecutionError`, not a task-quality failure. Its `error.message` gives a privacy-safe explanation and a next step. Check the local traceback for the failing tool, callback, model provider, or runtime configuration, fix the underlying issue, and rerun. A missing Harnest invocation context during managed evaluation is an integration error; do not weaken context checks or lower metric thresholds to work around it.
## Inspect in the playground
Run `harnest serve AGENT_DIR`, open `/`, and select **Evals**. The workspace lists root suites, configured and supported metrics, and the `business` and `strict` policies. You can run one suite and inspect case, metric, response, and tool-call evidence.
The playground response is optimized for interactive inspection. Use the CLI's `EvalRunResult` file when you need the complete retention contract.
For final-turn-only multi-turn metrics, the playground can currently mark a run as failed when earlier rows are `not_evaluated`, even if the CLI's scored-turn gate passes. Use the CLI quality gate for CI and inspect the final scored turn when diagnosing this difference.
## Use results in CI
```bash theme={null}
mkdir -p artifacts
harnest test AGENT_DIR --evals --no-output \
--eval-output artifacts/eval-result.json
```
The command exit status is the primary quality gate. You can also inspect a retained result after the command completes:
```bash theme={null}
jq -e '.apiVersion == "harnest.dev/v1alpha1" and .status == "passed"' \
artifacts/eval-result.json
```
Configure your CI artifact-upload step to run even when `harnest test` fails. Otherwise the most useful failure diagnostics may be discarded.
Complete results can contain prompts, responses, tool arguments and results, rubric rationales, user IDs, and session state. Treat the file as sensitive. Apply short retention, restricted access, and your normal secret and personal-data controls. Do not publish it as an unrestricted build artifact.
## Troubleshoot a run
| Symptom | Cause or next check |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--eval-output requires --evals` | Add `--evals`; no other test lane produces this result contract |
| No eval sets found | Add at least one root `evals/*.evalset.json`; `test_config.json` alone is invalid |
| Eval set ID mismatch | Match `eval_set_id` to the filename before `.evalset.json` |
| Result file is absent | Check whether compilation or Python tests failed before evaluation started |
| Result status is `error` | Read `error.type` and `error.message`, then verify model, service, and tool credentials |
| `EvaluationExecutionError` | Execution or scoring failed without metric results. Fix the underlying runtime error before judging task quality; if Harnest context is unavailable through the managed CLI or playground, report an integration bug |
| Ambiguous agent model transport | Select an eval model ID with one exact transport match, or remove conflicting same-provider transport choices; see [transport selection](/docs/harnest/build/models-and-libraries/configure-a-model#reuse-an-agents-model-client) |
| Metric is `not_evaluated` | Inspect its per-invocation details and required evidence; this status does not pass |
| Judge results vary | Increase `numSamples`, tighten the rubric, or use a deterministic reference metric |
| Unexpected model cost | Count the agent calls, simulated turns, judge metrics, and each metric's `numSamples` |
| LangGraph rejects input | Use non-empty text-only `userContent`; move media and live behavior to smoke tests |