Create a custom metric

Use a custom metric for rules that built-in scoring cannot express. This page integrates a scoring service; Create custom evals provides a complete local scorer, test cases, and unit tests. Custom metrics work with both ADK and LangGraph.

Harnest scoring helper

Create a bare scorer with harnest add eval company-quality --metric custom. It creates lib/company_quality.py without changing evaluation config. Implement the stub before registering it below. Use @metric for sync or async scoring functions. Native ADK scorers still work without the decorator and can share the same configuration. Add httpx to your dependencies and set EVAL_SERVICE_URL and EVAL_SERVICE_TOKEN. This example expects {"score": 0.9, "reason": "All required facts are present"}; adapt the payload to your service.

lib/company_evals.py

import os
import httpx
from harnest.evaluation import MetricContext, MetricScore, metric

@metric
async def company_quality(context: MetricContext) -> MetricScore:
    """Ask the team's service to score this conversation."""
    payload = {
        "actual": [turn.model_dump(mode="json", by_alias=True)
                   for turn in context.actual],
        "expected": None if context.expected is None else [
            turn.model_dump(mode="json", by_alias=True)
            for turn in context.expected
        ],
    }
    async with httpx.AsyncClient(timeout=20) as client:
        response = await client.post(
            os.environ["EVAL_SERVICE_URL"],
            headers={"Authorization": f"Bearer {os.environ['EVAL_SERVICE_TOKEN']}"},
            json=payload,
        )
        response.raise_for_status()
    verdict = response.json()
    return MetricScore(score=verdict["score"], evidence=verdict.get("reason"))

evals/test_config.json

{
  "criteria": {"company_quality": 0.85},
  "customMetrics": {
    "company_quality": {
      "codeConfig": {"name": "harnest.lib.company_evals.company_quality"}
    }
  }
}

harnest test ./my-agent --evals --eval-output artifacts/eval-results.json

MetricContext provides native ADK metric, actual and optional expected invocation tuples, optional scenario, and the configured threshold. Choose a MetricScore return policy: one conversation score, one score per turn, or an explicitly unscored result. Scores pass at or above the threshold. Use finite numbers; declare ranges outside 0–1 with metricInfo. Invalid results and service errors fail the evaluation. Evidence is saved as ADK rubric rationale in the result JSON. Use the CLI for CI; the playground only scores the final turn.

Use a native ADK scorer

Use the native scorer contract when an existing evaluator needs direct ADK inputs or result control. Both sync and async scorers are supported; use async I/O for network scoring. Actual and expected turn counts can differ, and simulations may have no golden response. Keep aggregate and turn-level scores consistent. The native result contract explains per-turn rows and unevaluated results; the custom evaluation walkthrough shows a complete implementation.

Register the function

Use the same name under criteria and customMetrics. Authored root lib/ modules compile below harnest.lib, so lib/eval_metrics.py becomes harnest.lib.eval_metrics during evaluation.

evals/test_config.json

{
  "criteria": {
    "required_location_terms": 1.0
  },
  "customMetrics": {
    "required_location_terms": {
      "description": "Every answer contains Paris and France.",
      "codeConfig": {
        "name": "harnest.lib.eval_metrics.required_location_terms"
      }
    }
  }
}

Scores default to a 0–1 range. Set metricInfo for another range; it describes scores without rescaling them.

{
  "customMetrics": {
    "required_location_terms": {
      "codeConfig": {
        "name": "harnest.lib.eval_metrics.required_location_terms"
      },
      "metricInfo": {
        "metricName": "required_location_terms",
        "description": "Product-specific verification score.",
        "metricValueInfo": {
          "interval": {
            "minValue": 0.0,
            "maxValue": 1.0
          }
        }
      }
    }
  }
}

ADK replaces metricInfo.metricName with the owning customMetrics map key when registering the metric. Keep the two names identical in authored JSON for clarity.

Design a reliable metric

Make the function deterministic unless model judgment is essential.
Read the threshold from metric.criterion so configuration remains the source of truth.
Return NOT_EVALUATED with no score when evidence is genuinely unavailable; do not silently pass. A known violation, such as a missing required answer, can instead receive an explicit failing score.
Preserve actual and expected invocations in per-turn results so failures remain diagnosable.
Keep network calls explicit and separately authenticated. Harnest’s transport sharing applies to declared judge and simulator models, not arbitrary SDK calls inside a custom function. Your function must configure and clean up any clients it creates. It can read the same process environment as the agent; it should never return credentials in diagnostic evidence.

A custom metric is trusted application code. Harnest imports and executes the configured function. Do not run an eval configuration or agent repository from an untrusted source.

Troubleshoot registration

Symptom	Check
Metric is not found	The name exists in both `criteria` and `customMetrics`
Function cannot import	The path starts with `harnest.lib` and matches the file and function names
Metric is `not_evaluated`	The function returned no score or its required evidence was absent
Result row count is rejected	A scored metric returned fewer or more per-turn rows than there are actual invocations
Report and command disagree	Overall score/status and the mean of scored turn results use different policies
LangGraph evidence differs	The adapter exposes public final text and neutral tool events, not native private framework state

Overview

Get started

Harnest Studio

Guides

API reference

Create a custom metric

Harnest scoring helper

Use a native ADK scorer

Register the function

Design a reliable metric

Troubleshoot registration

​Harnest scoring helper

​Use a native ADK scorer

​Register the function

​Design a reliable metric

​Troubleshoot registration

Harnest scoring helper

Use a native ADK scorer

Register the function

Design a reliable metric

Troubleshoot registration