Eval sets and configuration

An eval set groups related cases using ADK’s EvalSet schema for either framework. Harnest validates these files during compilation. Custom questions do not require a custom Python metric. Use this page to author cases with built-in scoring, or follow Create custom evals when you need to implement your own scoring rule.

Add an evaluation

harnest add eval answer-quality --i

Choose from 13 built-in metrics or a custom scorer. The picker labels each scoring backend: local comparison, LLM judge, or Vertex evaluation service. -i and --interactive also work.

harnest add eval answer-quality                              # response text similarity
harnest add eval tool-routing --metric tool_trajectory_avg_score
harnest add eval company-quality --metric custom             # Python stub only

Built-ins create evals/<name>.evalset.json and add missing criteria to the shared evals/test_config.json. Existing criteria and settings are preserved; shared criteria apply to sets without a configuration of their own. Edit the sample cases, thresholds, and rubrics before running harnest test . --evals. Scaffolding does not install dependencies or call a model. Custom scoring creates only lib/<name>.py; implement it and register the metric. Names use underscores in generated files. Use --project PATH when running outside the agent folder.

Save a Playground conversation

After a chat turn in local harnest serve, select Save as eval, choose an existing suite or create one, review the expected replies, then select Save eval. The case is written directly to the agent’s evals/ folder. Restart plain serve to run the updated suite in the Playground, or use harnest serve --reload to pick up changes automatically. Sessions without a writable local workspace offer Download eval instead.

Directory contract

AGENT_DIR/evals/

evals/
├── answer-quality.evalset.json
├── tool-routing.evalset.json
└── test_config.json

Rule	Contract
Location	Root `evals/` directory only
Eval set name	`<eval-set-id>.evalset.json`
ID match	`eval_set_id` must equal `<eval-set-id>`
Case IDs	Every `evalId` must be unique within its set
Layout	Flat; nested directories and unexpected public files are rejected
Order	Eval set filenames are sorted before execution
Configuration	Optional shared `test_config.json` and optional `<set>.test_config.json` per set

Nested SubAgent eval assets may be reached while validating composition, but harnest test --evals runs only the root agent’s eval sets.

Author a static case

A static case replays each authored userContent and records the actual agent response. The authored finalResponse and intermediateData are the golden values used by reference metrics.

evals/city-facts.evalset.json

{
  "eval_set_id": "city-facts",
  "name": "City fact answers",
  "description": "Checks a verified answer and a context-dependent follow-up.",
  "eval_cases": [
    {
      "evalId": "verify_paris",
      "conversation": [
        {
          "userContent": {
            "role": "user",
            "parts": [{"text": "Use get_city_fact to verify Paris's country and one landmark."}]
          },
          "finalResponse": {
            "role": "model",
            "parts": [{"text": "Paris is the capital of France, and its verified landmark is the Eiffel Tower."}]
          },
          "intermediateData": {
            "toolUses": [
              {"name": "get_city_fact", "args": {"city": "Paris"}}
            ],
            "toolResponses": []
          }
        },
        {
          "userContent": {
            "role": "user",
            "parts": [{"text": "Which country is it the capital of? Do not call the tool again."}]
          },
          "finalResponse": {
            "role": "model",
            "parts": [{"text": "It is the capital of France."}]
          },
          "intermediateData": {
            "toolUses": [],
            "toolResponses": []
          }
        }
      ],
      "sessionInput": {
        "appName": "city_facts",
        "userId": "eval-user",
        "state": {}
      }
    }
  ]
}

Both turns run in one session: “it” in the second prompt refers to Paris in the first. Keep the default history="session" for this test. See Agents and graphs. The expected behavior is explicit per turn:

Turn	Expected response	Expected tool behavior
1	Names Paris, France, and the verified landmark	Calls `get_city_fact(city="Paris")` once
2	Resolves “it” from history and answers France	Makes no tool call

Inspect per-turn content, tool events, and session evidence in Run and inspect evaluation results.

Case fields

Field	Required	Purpose
`evalId`	Yes	Stable case identity
`conversation`	One of two	Static invocation list
`conversationScenario`	One of two	Plan for a user simulator
`sessionInput`	No	App name, user ID, optional session ID, and initial state
`rubrics`	No	Rubrics inherited by the case’s invocations
`finalSessionState`	No	Authored expected final state retained by the ADK case model

Exactly one of conversation and conversationScenario is required.

Invocation fields

Field	Purpose
`userContent`	User input represented as Google GenAI content parts
`finalResponse`	Golden model response for reference-based metrics
`intermediateData.toolUses`	Golden ordered function calls, including arguments
`intermediateData.toolResponses`	Golden function responses when needed by a metric
`rubrics`	Rubrics scoped to this invocation

Scoring excludes provider-marked reasoning and retains visible text and tool events.

Score sets independently

To score tool_calls.evalset.json only on tool use, create a configuration with the same set name:

evals/tool_calls.test_config.json

{
  "criteria": {"tool_trajectory_avg_score": 1.0}
}

The CLI and Playground use this entire configuration for the matching set. Other sets use the shared evals/test_config.json unless they have a configuration of their own. All cases in a set use the same criteria. Split cases into different sets when they need different scoring. Use response_match_score when the wording of a reference answer matters. A longer, semantically correct answer can score poorly on text overlap. For open-ended answers, use a rubric or a judge metric from the metrics reference.

Configure criteria

Every key under criteria names a built-in or custom metric. A numeric value is a threshold. Metrics with judge, rubric, hallucination, simulation, or trajectory options use an object containing threshold and their additional fields.

evals/test_config.json

{
  "criteria": {
    "tool_trajectory_avg_score": {
      "threshold": 1.0,
      "matchType": "IN_ORDER"
    },
    "response_match_score": 0.8,
    "final_response_match_v2": {
      "threshold": 0.8,
      "judgeModelOptions": {
        "numSamples": 3,
        "parallelismLimit": 1
      }
    }
  }
}

If neither configuration exists for a set, ADK uses these default criteria:

{
  "criteria": {
    "tool_trajectory_avg_score": 1.0,
    "response_match_score": 0.8
  }
}

A threshold is a pass boundary, not a target shown only in reports. A metric passes when its score is greater than or equal to its threshold. A failed metric makes its case and the command fail.

Choose tool trajectory behavior

The trajectory policy affects only tool_trajectory_avg_score, preserving its threshold.

Policy	Effective ADK match type	Behavior
`business`	`IN_ORDER`	Every expected call and argument must occur in order. Extra discovery or helper calls are allowed.
`strict`	`EXACT`	Actual and expected calls, order, and arguments must match exactly.

harnest test AGENT_DIR --evals
harnest test AGENT_DIR --evals --eval-trajectory strict

The CLI selection replaces an authored matchType for this metric. Use separate business and strict CI lanes when you need both behavioral confidence and exact orchestration.

Add rubrics

Rubric metrics require at least one testable property. The simplest reusable location is the metric criterion in test_config.json:

{
  "rubricId": "grounded_city_answer",
  "rubricContent": {
    "textProperty": "The response names Paris and accurately states that it is the capital of France."
  }
}

Rubrics can also live on an eval case or static invocation. Set their type to FINAL_RESPONSE_QUALITY, TOOL_USE_QUALITY, or TRAJECTORY_QUALITY so ADK routes them to the intended built-in rubric evaluator. Keep rubricId values unique within the effective set. Criterion-level rubrics do not need a type. See Metrics reference for judge configuration and score behavior.

Overview

Get started

Harnest Studio

Guides

API reference

Eval sets and configuration

Add an evaluation

Save a Playground conversation

Directory contract

Author a static case

Case fields

Invocation fields

Score sets independently

Configure criteria

Choose tool trajectory behavior

Add rubrics

​Add an evaluation

​Save a Playground conversation

​Directory contract

​Author a static case

​Case fields

​Invocation fields

​Score sets independently

​Configure criteria

​Choose tool trajectory behavior

​Add rubrics

Add an evaluation

Save a Playground conversation

Directory contract

Author a static case

Case fields

Invocation fields

Score sets independently

Configure criteria

Choose tool trajectory behavior

Add rubrics