Accept multimodal data

Define a Pydantic contract once. Harnest applies it to agents, tools, JSON, SSE, WebSockets, SubAgents, and session messages.

Choose a content type

Content	Type	Configure with
Text	`Text`	Pydantic field rules
Images	`Image`	`ImageConstraints`
Audio	`Audio`	`AudioConstraints`
Video	`Video`	`VideoConstraints`
Files	`File`	`FileConstraints`
Custom JSON	`Data[T]`	`DataConstraints`
Generic stored media	`AssetRef`	Pydantic field rules

Media constraints can limit MIME type, decoded byte size, dimensions, pixels, duration, pages, frame rate, sample rate, channels, animation, and archive expansion.

Define the contract

Keep shared contracts in the root models/ directory. Use Annotated to keep limits beside the field they protect.

models/vision.py

from typing import Annotated

from pydantic import BaseModel

from harnest.content import Image, ImageConstraints


Screenshot = Annotated[
    Image,
    ImageConstraints(
        media_types=frozenset({"image/jpeg", "image/png"}),
        max_bytes=500 * 1024,
        max_width=1920,
        max_height=1080,
        max_pixels=2_073_600,
        animated=False,
    ),
]


class VisionInput(BaseModel):
    question: str
    screenshot: Screenshot


class VisionOutput(BaseModel):
    answer: str

Harnest inspects decoded bytes. Client-supplied metadata does not bypass the contract.

Apply the contract

Agent input and output
Server tool output
Client tool output

agent.py

from harnest.agent import Agent
from harnest.model import LiteLLMModel
from harnest.models.vision import VisionInput, VisionOutput


root_agent = Agent(
    name="vision",
    model=LiteLLMModel("provider/vision-model"),
    input_schema=VisionInput,
    output_schema=VisionOutput,
)

tools/analyze.py

from harnest.models.vision import VisionOutput
from harnest.agent import tool


@tool(output_schema=VisionOutput)
async def analyze() -> VisionOutput:
    return VisionOutput(answer="Ready")

tools/browser.py

from pydantic import BaseModel

from harnest.agent import client_tool
from harnest.models.vision import Screenshot


class CaptureResult(BaseModel):
    screenshot: Screenshot


@client_tool(output_schema=CaptureResult)
async def capture_viewport() -> CaptureResult:
    """Capture the visible browser viewport."""
    ...

Client-tool stubs execute on the client. Harnest validates the submitted result before the waiting agent or SubAgent resumes.

Send transient media

Without a storage annotation, media is available only to the current model turn.

{
  "sessionId": "demo",
  "input": {
    "question": "What is visible?",
    "screenshot": {
      "type": "image",
      "mediaType": "image/jpeg",
      "data": "<base64>"
    }
  }
}

Harnest validates the bytes

The Pydantic field limits are applied to inspected content.

The adapter builds the model request

ADK or LangGraph receives the media only for the immediate provider call. This includes mid-turn tool results inside SubAgents.

Harnest clears the private lease

Success clears the bytes. A provider retry can reuse them before that point.

Surface	Stored value
Native checkpoint and history	`{ "type": "image", "mediaType": "image/jpeg", "content": "attached" }`
Logs, traces, and audit records	No bytes or private correlation IDs
Public events and `/messages`	Safe attachment placeholder
Final inline agent output	Returned once on the authenticated response; not replayable

Store and retrieve media

Return durable references or make media replayable.

Build client tools

Resume an agent with browser or device data.

Overview

Get started

Guides

Accept multimodal data

Choose a content type

Define the contract

Apply the contract

Send transient media

Store and retrieve media

Build client tools

​Choose a content type

​Define the contract

​Apply the contract

​Send transient media

Store and retrieve media

Build client tools

Choose a content type

Define the contract

Apply the contract

Send transient media