LLM Graders

LLM graders use a language model to evaluate agent responses against custom criteria defined in a prompt file.

Explicit LLM Graders

Put semantic grading requirements in assert. Plain strings are handled by the built-in llm-rubric rubric grader. Use type: llm-rubric when you need a custom prompt, provider, or grader-specific transform:

prompts:
  - "{{ task }}"

tests:
  - id: simple-eval
    vars:
      task: "Debug this function..."
    assert:
      - Correctly explains the bug and proposes a fix

Reference answers belong in vars.expected_output. That var is available to graders, but it does not create an LLM grading call by itself. Consume it from an explicit assertion, usually type: llm-rubric with a value such as "Matches the reference answer: {{ expected_output }}". See How reference fields and assertions interact.

Configuration

Reference an LLM grader in your eval file:

assert:
  - name: semantic_check
    type: llm-rubric
    prompt: file://graders/correctness.md
    provider: grader_gpt_5_mini   # optional: route this grader to a named LLM provider

Use provider: when you want different llm-rubric entries in the same eval to run on different grader models. This is useful for grader panels, majority-vote ensembles, and grader A/B benchmarks.

Agent Rubrics

Use type: agent-rubric when a Promptfoo-style rubric needs an agentic grader that can inspect the workspace instead of only judging the final answer text. AgentV routes agent-rubric through the same shared LLM rubric scoring path, so results still appear as normal EvaluationScore entries.

assert:
  - name: workspace-evidence
    type: agent-rubric
    provider: codex-grader
    value: The answer's claims are backed by concrete files in the workspace.

The resolved grader provider must be agent-capable, such as a Codex, Claude, Copilot, Pi, VS Code, or agentv provider. If provider: resolves to a plain LLM grader provider, AgentV fails clearly instead of silently downgrading the check.

For agent-backed grading, AgentV creates a temporary verdict.json path for each grading call and instructs the agent grader to write exactly one JSON object there:

{ "pass": true, "score": 1, "reason": "Evidence found in src/logger.ts." }

AgentV reads that file before parsing the final assistant message. The score must be a finite number from 0 to 1; malformed verdict files fail closed as grader failures. If the file is missing, AgentV may still parse the final assistant text using the same structured JSON fallback used by other agent grader modes.

Prompt Files

The prompt file defines evaluation criteria and scoring guidelines. It can be a markdown text template or a TypeScript/JavaScript dynamic template.

Markdown Template

Write evaluation instructions as markdown. Template variables are interpolated:

# Evaluation Criteria

Evaluate the candidate's response to the following question:

**Question:** {{input}}
**Criteria:** {{criteria}}
**Reference Answer:** {{expected_output}}
**Candidate Answer:** {{output}}

## Scoring

Score the response from 0.0 to 1.0 based on:
1. Correctness — does the output match the expected outcome?
2. Completeness — does it address all parts of the question?
3. Clarity — is the response clear and well-structured?

Available Template Variables

Variable	Source
criteria	Test criteria field
input	Resolved input text
expected_output	Reference answer text
output	Candidate answer text
metadata	Test metadata as formatted JSON
metadata_json	Test metadata as compact JSON
rubric	Rubric data as structured JSON when available, or criteria text otherwise
rubrics	llm-rubric rubric items as formatted JSON
rubrics_json	llm-rubric rubric items as compact JSON
file_changes	Unified diff of workspace file changes (populated when workspace is configured)
tool_calls	Formatted summary of tool calls from agent execution (tool name + key inputs per call)

Use prompt: ./path/to/prompt.md for the common relative-path case. Use prompt: file://path/to/prompt.md only when you need to force file-reference resolution explicitly.

Structured task input belongs in input. If input is a message whose content is a JSON object, {{input}} renders that object as formatted JSON for the grader prompt; no separate grader-only input field is required. Use metadata for provenance or suite-level source fields, and rubrics_json for rubric arrays.

For Promptfoo-compatible custom rubric prompts, the standard judge prompt uses {{ output }} for the candidate answer and {{ rubric }} for the rendered assertion value. The judge response should be JSON-compatible with { "reason": string, "pass": boolean, "score": number }; AgentV normalizes that into public grading.json fields reason, pass, and score.

Suite-level metadata is inherited by every test. When rubric items vary per test, keep the grader on each test and reuse the prompt file:

metadata:
  source_repo: https://github.com/virattt/dexter
  source_commit: 8d9419829f443f84b804d033bb2c3b1fbd788629
  source_file: src/evals/dataset/finance_agent.csv

prompts:
  - "{{ input }}"

tests:
  - id: apple-research
    vars:
      input:
        company: Apple
        ticker: AAPL
    metadata:
      row: 1
    assert:
      - name: dexter_semantic
        type: llm-rubric
        prompt: file://prompts/dexter-grader.md
        value:
          - operator: correctness
            outcome: Uses the provided ticker and company.

Per-Grader Provider

By default, an llm-rubric uses tests[].options.provider, then default_test.options.provider, then defaults.grader from the resolved config graph. Each value selects from the same providers pool used for candidate providers; AgentV does not infer a grader from the first candidate provider. Override the provider per assertion when you need multiple grader models in one run:

default_test:
  options:
    provider: grader_gpt_5_mini

tests:
  - id: strict-case
    options:
      provider: grader_claude_haiku
    assert:
      - name: semantic_quality
        type: llm-rubric
        value: The answer is correct and concise.

assert:
  - name: grader-gpt
    type: llm-rubric
    provider: grader_gpt_5_mini
    prompt: ./prompts/pass-fail.md
  - name: grader-haiku
    type: llm-rubric
    provider: grader_claude_haiku
    prompt: ./prompts/pass-fail.md

Each provider: value must match a provider label or backend id from .agentv/config.yaml or from an eval-local providers entry.

TypeScript Template

For dynamic prompt generation, use the definePromptTemplate function from agentv:

#!/usr/bin/env bun
import { definePromptTemplate } from 'agentv';

function textFromMessages(messages: Array<{ content?: unknown }>): string {
  return messages
    .map((message) => typeof message.content === 'string' ? message.content : '')
    .filter(Boolean)
    .join('\n');
}

export default definePromptTemplate((ctx) => {
  const rubric = ctx.config?.rubric as string | undefined;
  const question = textFromMessages(ctx.input.filter((message) => message.role === 'user'));
  const referenceAnswer = textFromMessages(ctx.expectedOutput);
  const candidateAnswer = ctx.output ?? '';

  return `You are evaluating an AI assistant's response.

## Question
${question}

## Candidate Answer
${candidateAnswer}

${referenceAnswer ? `## Reference Answer\n${referenceAnswer}` : ''}

${rubric ? `## Evaluation Criteria\n${rubric}` : ''}

Evaluate and provide a score from 0 to 1.`;
});

How It Works

1. AgentV renders the prompt template with variables from the test
2. The rendered prompt is sent to the selected grader target
3. The LLM returns a structured evaluation with score, assertions array, and reasoning
4. Results are recorded in the output JSONL

Command Configuration

When using TypeScript templates, configure them in YAML with optional config data passed to the command:

assert:
  - name: custom-eval
    type: llm-rubric
    prompt:
      command: [bun, run, ../prompts/custom-grader.ts]
      config:
        rubric: "Your rubric here"
        strictMode: true

The config object is available as ctx.config inside the template function.

Transforming File Outputs

If an agent returns a ContentFile block instead of plain text, use transform to convert that file into text before llm-rubric builds the candidate prompt.

AgentV always tries a default UTF-8 text read first. That is enough for text-based formats such as CSV, JSON, SQL, Markdown, YAML, HTML, XML, and plain text. For binary formats such as .xlsx, .pdf, or .docx, add a transform:

default_test:
  options:
    transform: file://scripts/transforms/xlsx-to-csv.ts

tests:
  - id: spreadsheet-output
    vars:
      input: Generate the spreadsheet report
    assert:
      - Output includes the revenue rows
      - name: spreadsheet-check
        type: llm-rubric
        prompt: |
          Check whether the transformed spreadsheet text contains the revenue rows:

          {{ output }}

Resolution order:

• default_test.options.transform applies to every test unless overridden
• tests[].options.transform overrides default_test.options.transform for that test
• assertion-level transform applies to that grader only, after the test/default transform
• if no transform is configured, AgentV falls back to a UTF-8 text read
• if the fallback read looks binary or invalid, the grader receives a warning note instead of failing the test run

See examples/features/file-transforms/ for a runnable example with a file-producing target and a spreadsheet conversion script.

Available Context Fields

TypeScript templates receive a context object with these fields:

Field	Type	Description
input	Message[]	Full resolved input messages
output	string | null	Candidate final answer / scored result
answer	string	Same final answer string, exposed for ergonomic handler code
messages	Message[]	Transcript messages from the target execution
criteria	string	Test criteria field
expectedOutput	Message[]	Full resolved expected output
trace	Trace	Full execution trace with messages, events, metrics, and provenance
traceSummary	TraceSummary	Lightweight execution metrics summary
metadata	object	Test metadata after suite defaults are merged
config	object	Custom config from YAML

The raw prompt-template stdin uses snake_case keys such as expected_output, trace_summary, and token_usage. definePromptTemplate() converts them to SDK camelCase fields before calling your handler.

Template Variable Derivation

Template variables are derived internally through three layers:

1. Authoring Layer

What users write in YAML or JSONL:

• Prompt templates render from tests[].vars and default_test.vars.
• vars.expected_output is a conventional reference-answer var. It is available to assertion values and grader templates, but it is not an authored sibling field on the test.

2. Resolved Layer

After prompt rendering, canonical message arrays represent the resolved prompt input and reference data:

• input: TestMessage[] — canonical resolved input
• expected_output: TestMessage[] — canonical resolved expected output

At this layer, authored prompt vars and resolved grader context are separate.

3. Template Variable Layer

Derived strings injected into grader prompts:

Variable	Derivation
criteria	Passed through from the test field
input	Resolved input text
expected_output	Reference answer text
output	Candidate answer text
metadata_json	Test metadata, compact JSON
file_changes	Unified diff of workspace file changes (populated when workspace is configured)
tool_calls	Formatted summary of tool calls from agent execution (tool name + key inputs per call)

Example flow:

# User writes:
prompts:
  - "{{ question }}"
tests:
  - vars:
      question: "What is 2+2?"
      expected_output: "The answer is 4"
    assert:
      - type: llm-rubric
        value: "Matches the reference answer: {{ expected_output }}"

# Resolved:
input:    [{ role: "user", content: "What is 2+2?" }]
expected_output: [{ role: "assistant", content: "The answer is 4" }]

# Derived template variables:
input:           "What is 2+2?"
expected_output: "The answer is 4"
output:          (extracted from provider output at runtime)