Metrics reference

Both frameworks use Google ADK 2.x for scoring. Available metrics come from the installed ADK registry. The 13 metrics below ship with ADK 2.8, Harnest’s minimum version. Later compatible releases may add metrics; check the playground’s Evals catalog for your installed list.

Built-in metrics

Reference and single-turn metrics

Metric	What it measures	Score	Needs golden result	Backend
`tool_trajectory_avg_score`	Tool names and arguments across each invocation	0–1	Tool calls	Local deterministic comparison
`response_match_score`	ROUGE-1 similarity between actual and golden final text	0–1	Final response	Local deterministic comparison
`response_evaluation_score`	Coherence of the actual response	1–5	No	Vertex Gen AI evaluation service
`safety_v1`	Harmlessness of the actual response	0–1	No	Vertex Gen AI evaluation service
`final_response_match_v2`	Meaning-level validity against the golden final response	0–1	Final response	Configured ADK judge model
`rubric_based_final_response_quality_v1`	Final answer behavior against authored rubrics	0–1	Rubrics	Configured ADK judge model
`hallucinations_v1`	Fraction of claims supported by available context and evidence	0–1	No	Configured ADK judge model
`rubric_based_tool_use_quality_v1`	Actual tool use against authored rubrics	0–1	Rubrics	Configured ADK judge model

tool_trajectory_avg_score gives each invocation either 0 or 1, then averages the invocation scores. Choose business or strict to control matching. final_response_match_v2 produces a binary verdict for each invocation. It uses majority vote across judge samples, then reports the fraction of valid invocations.

Multi-turn and simulation metrics

Metric	What it measures	Score	Backend
`per_turn_user_simulator_quality_v1`	Fraction of simulated user messages that follow the conversation plan	0–1	Configured ADK judge model
`multi_turn_task_success_v1`	Whether the agent achieves the conversation’s goals	0–1	Vertex Gen AI evaluation service
`multi_turn_trajectory_quality_v1`	Quality of the full path used to achieve the goals	0–1	Vertex Gen AI evaluation service
`multi_turn_tool_use_quality_v1`	Function-call quality across the conversation	0–1	Vertex Gen AI evaluation service
`rubric_based_multi_turn_trajectory_quality_v1`	Cumulative responses and tool behavior against authored rubrics	0–1	Configured ADK judge model

The three Vertex multi-turn metrics are reference-free and evaluate the complete actual conversation. Their per-invocation result is not_evaluated on earlier turns; the last turn carries the conversation-level score. The rubric-based multi-turn metric uses the same final-turn convention.

Configure thresholds

Use a number for a metric that needs only a threshold:

evals/test_config.json

{
  "criteria": {
    "response_match_score": 0.8,
    "response_evaluation_score": 3.0,
    "safety_v1": 0.9,
    "multi_turn_task_success_v1": 0.8,
    "multi_turn_trajectory_quality_v1": 0.8,
    "multi_turn_tool_use_quality_v1": 0.8
  }
}

Scores equal to the threshold pass. Choose a threshold inside the metric’s documented score interval.

Configure model-judged metrics

Configure model-judged metrics through their criteria: Verdict parsing uses the judge’s final response, including any rationale. Reasoning and partial chunks are excluded; a missing final answer cannot pass. includeIntermediateResponsesInFinal instead controls evidence from the agent under test. See the judge configuration reference for thresholds, model overrides, sampling, concurrency, and intermediate-response settings. Higher sample counts increase judge calls and cost.

evals/test_config.json

{
  "criteria": {
    "final_response_match_v2": {
      "threshold": 0.8,
      "judgeModelOptions": {
        "numSamples": 3,
        "parallelismLimit": 1
      }
    },
    "hallucinations_v1": {
      "threshold": 0.9,
      "evaluateIntermediateNlResponses": false
    }
  }
}

hallucinations_v1 uses evaluateIntermediateNlResponses, not includeIntermediateResponsesInFinal, to opt visible intermediate natural-language responses into its analysis.

Use a custom judge model

Model precedence is an explicit judgeModel, then the required OPENAI_MODEL. There is no built-in model fallback. Use an explicit provider-prefixed model ID when the judge should differ from the agent:

evals/test_config.json

{
  "criteria": {
    "final_response_match_v2": {
      "threshold": 0.8,
      "judgeModelOptions": {
        "judgeModel": "openai/my-judge-model",
        "numSamples": 3
      }
    }
  }
}

Replace my-judge-model with a model served by your endpoint. The judge reuses a compatible agent transport when available. See model environment for endpoint and credential setup.

Harnest runs each eval set once. That num_runs=1 safeguard does not replace judgeModelOptions.numSamples: a judge-backed metric can still call its judge several times for one invocation. Lower the sample count deliberately when cost matters, and raise it when judge stability matters more.

Configure rubric metrics

Each rubric criterion requires a non-empty rubrics list. Every rubric has a stable ID and one testable text property.

evals/test_config.json

{
  "criteria": {
    "rubric_based_final_response_quality_v1": {
      "threshold": 1.0,
      "judgeModelOptions": {
        "numSamples": 3
      },
      "rubrics": [
        {
          "rubricId": "grounded_answer",
          "rubricContent": {
            "textProperty": "The answer states only claims supported by tool evidence."
          }
        }
      ]
    },
    "rubric_based_tool_use_quality_v1": {
      "threshold": 1.0,
      "judgeModelOptions": {
        "numSamples": 3
      },
      "rubrics": [
        {
          "rubricId": "verify_first",
          "rubricContent": {
            "textProperty": "The agent calls get_city_fact before making a factual claim."
          }
        }
      ]
    }
  }
}

Use rubric_based_multi_turn_trajectory_quality_v1 with the same criterion shape when a property spans the complete conversation. Rubric verdicts, scores, and rationales are preserved in the complete CLI result.

Configure evaluation backends

The agent under test and every metric backend inherit the same process environment. Follow Configure a model for local, CI, and deployment setup. Metrics may add a second provider to the agent’s normal model and tool stack:

Local deterministic
ADK judge model
Vertex evaluation service

tool_trajectory_avg_score and response_match_score need no metric service. The agent itself can still make live model or tool calls.

Omit judgeModel to use OPENAI_MODEL and the shared OPENAI_BASE_URL and optional OPENAI_API_KEY configuration. Choose these settings before running live evaluations. An explicit judgeModel, such as gemini-2.5-flash, opts into that ADK model provider and requires its own credentials. Harnest does not replace explicit model IDs.

Hosted scoring uses Google ADK 2.13 or newer with the Vertex AI SDK 2.7 or newer in the 2.x series. If your agent pins an earlier ADK release, update the framework lock and run harnest env sync . --profile eval before selecting these metrics.response_evaluation_score, safety_v1, and the three multi_turn_*_v1 metrics call the Vertex Gen AI evaluation client. If GOOGLE_API_KEY is set, ADK constructs that client with the key. Otherwise set both GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION and make Application Default Credentials available. When all are set, the API key path takes precedence.

test_config.json selects metric backends and judge model IDs. Never put credentials in that file.

Overview

Get started

Harnest Studio

Guides

API reference

Metrics reference

Built-in metrics

Reference and single-turn metrics

Multi-turn and simulation metrics

Configure thresholds

Configure model-judged metrics

Use a custom judge model

Configure rubric metrics

Configure evaluation backends

​Built-in metrics

​Reference and single-turn metrics

​Multi-turn and simulation metrics

​Configure thresholds

​Configure model-judged metrics

​Use a custom judge model

​Configure rubric metrics

​Configure evaluation backends

Built-in metrics

Reference and single-turn metrics

Multi-turn and simulation metrics

Configure thresholds

Configure model-judged metrics

Use a custom judge model

Configure rubric metrics

Configure evaluation backends