Simulations and live evaluation

A conversation scenario describes a goal instead of scripting every user turn. An ADK user simulator sends the fixed first prompt, generates follow-up messages from the plan, and stops when the task is complete or the turn limit is reached. Text scenarios work with ADK and LangGraph. Audio simulation and liveModelConfig are ADK-only.

Author a conversation scenario

evals/city-facts-simulation.evalset.json

{
  "eval_set_id": "city-facts-simulation",
  "name": "Simulated city fact conversation",
  "eval_cases": [
    {
      "evalId": "simulated_paris_fact_finding",
      "conversationScenario": {
        "startingPrompt": "I would like one verified fact about Paris.",
        "conversationPlan": "Ask the agent to verify a fact about Paris. Make sure it identifies France as the country whose capital is Paris. Once it has done so, end the conversation. Do not ask about another city."
      },
      "sessionInput": {
        "appName": "city_facts",
        "userId": "simulated-eval-user",
        "state": {}
      }
    }
  ]
}

Write the plan as observable user intent. Include completion conditions, allowed follow-ups, and actions the simulator must avoid. The simulator receives conversation history on each turn.

Configure text simulation

evals/test_config.json

{
  "criteria": {
    "per_turn_user_simulator_quality_v1": {
      "threshold": 0.8,
      "stopSignal": "</finished>",
      "judgeModelOptions": {
        "numSamples": 1
      }
    }
  },
  "userSimulatorConfig": {
    "type": "llm_backed",
    "maxAllowedInvocations": 4,
    "includeFunctionCalls": true
  }
}

Property	Default	Purpose
`type`	Legacy fallback is `llm_backed`	Selects the simulator; write it explicitly
`model`	Configured `OPENAI_MODEL`	Generates subsequent user turns; an explicit ADK model ID overrides the shared model
`modelConfiguration`	ADK default generation config	Google GenAI generation options for the simulator
`maxAllowedInvocations`	`20`	Stops a runaway conversation; includes the fixed starting prompt
`includeFunctionCalls`	`false`	Includes tool calls and responses in history shown to the simulator
`customInstructions`	ADK’s built-in prompt	Replaces the simulator prompt template

A custom instruction template must contain {{ stop_signal }}, {{ conversation_plan }}, and {{ conversation_history }}. Include {{ persona }} if the scenario uses a user persona. Harnest validates this through ADK before running the eval. An explicit userSimulatorConfig.model overrides OPENAI_MODEL. Otherwise, simulators use the shared model configuration, even when userSimulatorConfig is omitted. Compatible agent transports are reused; native providers need their own credentials. per_turn_user_simulator_quality_v1 is optional, but it is the metric designed to verify that generated user messages followed the scenario. Its stopSignal should match the simulator stop signal; </finished> is the shared default.

Set a finite maxAllowedInvocations. ADK accepts -1 for no limit, but an agent and simulator can otherwise continue indefinitely and spend unbounded model capacity.

Configure audio simulation

ADK 2.8 also exposes the llm_audio simulator. It first generates a text turn, then synthesizes audio through audioModel.

evals/test_config.json

{
  "criteria": {
    "per_turn_user_simulator_quality_v1": {
      "threshold": 0.8,
      "stopSignal": "</finished>",
      "judgeModelOptions": {
        "numSamples": 1
      }
    }
  },
  "userSimulatorConfig": {
    "type": "llm_audio",
    "maxAllowedInvocations": 4,
    "audioModel": "cloud_tts",
    "audioModelConfiguration": {
      "speechConfig": {
        "voiceConfig": {
          "prebuiltVoiceConfig": {"voiceName": "en-US-Studio-O"}
        },
        "languageCode": "en-US"
      }
    },
    "includeTextWithAudio": true
  }
}

Property	Default	Purpose
`audioModel`	`cloud_tts`	Google Cloud TTS adapter or another audio model ID resolved by ADK
`audioModelConfiguration`	US English Studio voice	Voice and response modality configuration
`includeTextWithAudio`	`true`	Emits text next to the synthesized audio part

cloud_tts uses Application Default Credentials. GOOGLE_CLOUD_PROJECT is used as its quota project when present. If you select a native model such as a Gemini TTS model, provide that model provider’s credentials and set its response modalities to audio.

Harnest passes audio simulation through to native ADK evaluation. LangGraph evaluation rejects the resulting non-text user content even when includeTextWithAudio is true, so it cannot silently score a text-only projection of an audio test.

Select ADK live inference

liveModelConfig changes the inference path used for the agent under test. Use it for an ADK agent configured with a bidirectional live model.

evals/test_config.json

{
  "criteria": {
    "safety_v1": 0.9
  },
  "liveModelConfig": {
    "timeoutSeconds": 300
  }
}

timeoutSeconds is the wait limit for live model turn completion and defaults to 300.

LangGraph --evals rejects any non-null liveModelConfig. Use text evals for portable behavior and an opted-in smoke test for LangGraph bidirectional media.

Configure models and services

Simulation can involve three independently authenticated components. They all inherit the environment of the Harnest command:

Component	Configuration	Verified setup
Agent under test	`LiteLLMModel.from_openai_environment()`	Shared `OPENAI_MODEL`, `OPENAI_BASE_URL`, and optional `OPENAI_API_KEY`
Text simulator and per-turn judge	Omitted `model` and `judgeModel`	Same compatible API environment; explicit native provider IDs require their own credentials
Cloud audio synthesis	`audioModel: "cloud_tts"`	Enable Google Cloud Text-to-Speech and provide Application Default Credentials

Simulation makes live, potentially paid calls. Follow Configure a model and keep credentials out of eval JSON. Compatible API keys do not authenticate Cloud TTS or native Google evaluation services.

Overview

Get started

Harnest Studio

Guides

API reference

Simulations and live evaluation

Author a conversation scenario

Configure text simulation

Configure audio simulation

Select ADK live inference

Configure models and services

​Author a conversation scenario

​Configure text simulation

​Configure audio simulation

​Select ADK live inference

​Configure models and services

Author a conversation scenario

Configure text simulation

Configure audio simulation

Select ADK live inference

Configure models and services