Run and inspect evaluation results

By default, harnest test --evals compiles the agent, runs the selected Python test lanes, evaluates every valid root eval set, and returns a nonzero status when the quality gate fails. To run a team-owned evaluation framework alone or alongside ADK, configure runner packages.

Run evaluations

harnest test AGENT_DIR --evals

The unit lane runs first. Add --smoke to run smoke tests before evaluation:

harnest test AGENT_DIR --smoke --evals

A failed Python lane stops later lanes. Scored failures do not stop other eval sets from running. Suites run once, in filename order. Judge metrics may make multiple calls according to numSamples. ADK evaluations have Harnest invocation identity, skills, assigned sandboxes, resources, and credentials. ADK owns the evaluation runner, so lifecycle and plugin behavior can differ from serving. Context expires when the invocation ends. Evaluation sessions are in-memory and separate from deployed sessions. The CLI owns its evaluation resources; the playground reuses serving resources. The CLI ADK runtime does not start task workers or cron schedules—use smoke or integration tests for queued workflows. Tools and external services can still have real side effects.

For native ADK workflow roots that use its legacy runner, such as SequentialAgent or a non-LlmAgent BaseAgent, Harnest rejects a plugin’s before_run_callback that returns replacement Content. Interrupting that response stream can skip invocation cleanup. Return None from the callback and produce the response from the agent or a tool instead. This restriction does not apply to the normal LlmAgent/node execution path.

Select the trajectory policy

# Required business calls in order; extra helper calls are allowed.
harnest test AGENT_DIR --evals --eval-trajectory business

# Tool names, order, arguments, and call count must match exactly.
harnest test AGENT_DIR --evals --eval-trajectory strict

business is the default. The flag affects only tool_trajectory_avg_score.

Save the complete result

Without --no-output, Harnest prints detailed evaluator output and a complete structured JSON EvalRunResult after evaluation. Select a machine-readable file instead of parsing mixed terminal output:

harnest test AGENT_DIR --evals \
  --eval-output artifacts/eval-result.json

Harnest creates parent directories and atomically replaces the result file. For quiet CI output, combine the explicit file with --no-output:

harnest test AGENT_DIR --evals --no-output \
  --eval-output artifacts/eval-result.json

--no-output suppresses unit, smoke, evaluator, and final result output in the terminal. It does not suppress an explicitly selected result file. --eval-output requires --evals.

Understand `EvalRunResult`

Check apiVersion, overall status, and scored-turn evidence. error indicates execution or scoring failure; an entirely unscored metric does not pass. See the EvalRunResult envelope reference. Infrastructure failures may leave partial results. Failures before evaluation starts may leave no file. Fix the runtime issue before assessing task quality.

Inspect in the playground

Run harnest serve AGENT_DIR, open /, and select Evals. The workspace lists root suites, configured and supported metrics, and the business and strict policies. You can run one suite and inspect case, metric, response, and tool-call evidence. Use the CLI result file for complete results; the playground is designed for interactive inspection.

For final-turn-only multi-turn metrics, the playground can currently mark a run as failed when earlier rows are not_evaluated, even if the CLI’s scored-turn gate passes. Use the CLI quality gate for CI and inspect the final scored turn when diagnosing this difference.

Use results in CI

mkdir -p artifacts
harnest test AGENT_DIR --evals --no-output \
  --eval-output artifacts/eval-result.json

The command exit status is the primary quality gate. You can also inspect a retained result after the command completes:

jq -e '.apiVersion == "harnest.dev/v1alpha1" and .status == "passed"' \
  artifacts/eval-result.json

Configure your CI artifact-upload step to run even when harnest test fails. Otherwise the most useful failure diagnostics may be discarded.

Complete results can contain prompts, responses, tool arguments and results, rubric rationales, user IDs, and session state. Treat the file as sensitive. Apply short retention, restricted access, and your normal secret and personal-data controls. Do not publish it as an unrestricted build artifact.

Troubleshoot a run

Symptom	Cause or next check
`--eval-output requires --evals`	Add `--evals`; no other test lane produces this result contract
No eval sets found	Add at least one root `evals/*.evalset.json`; `test_config.json` alone is invalid
Eval set ID mismatch	Match `eval_set_id` to the filename before `.evalset.json`
Result file is absent	Check whether compilation or Python tests failed before evaluation started
Result status is `error`	Read `error.type` and `error.message`, then verify model, service, and tool credentials
`EvaluationExecutionError`	Execution or scoring failed without metric results. Fix the underlying runtime error before judging task quality; if Harnest context is unavailable through the managed CLI or playground, report an integration bug
Ambiguous agent model transport	Select an eval model ID with one exact transport match, or remove conflicting same-provider transport choices; see transport selection
Metric is `informational`	Inspect the measurement, such as tool-call count or token usage. It has no threshold and does not affect whether scored criteria pass.
Metric is `not_evaluated`	Inspect its per-invocation details and required evidence; this status does not pass
Judge results vary	Increase `numSamples`, tighten the rubric, or use a deterministic reference metric
Unexpected model cost	Count the agent calls, simulated turns, judge metrics, and each metric’s `numSamples`
LangGraph rejects input	Use non-empty text-only `userContent`; move media and live behavior to smoke tests

Overview

Get started

Harnest Studio

Guides

API reference

Run and inspect evaluation results

Run evaluations

Select the trajectory policy

Save the complete result

Understand `EvalRunResult`

Inspect in the playground

Use results in CI

Troubleshoot a run

​Run evaluations

​Select the trajectory policy

​Save the complete result

​Understand EvalRunResult

​Inspect in the playground

​Use results in CI

​Troubleshoot a run

Run evaluations

Select the trajectory policy

Save the complete result

Understand `EvalRunResult`

Inspect in the playground

Use results in CI

Troubleshoot a run