CLI Provider

The cli provider runs an arbitrary shell command per test case and captures its output as the target’s response. It’s the escape hatch that lets you evaluate anything that exposes a command-line entry point — your own agent, a third-party CLI, a stub that prints a fixed answer, a script that calls an in-house microservice, etc.

Because the contract is “we invoke a command and read a file,” almost any useful composition pattern (sanity-checking your grader against a known-good answer, diffing two implementations, calling an in-house queue) can be built on top without any new primitives.

Minimal example

.agentv/config.yaml

providers:
  - id: cli
    label: my-agent
    runtime: host
    config:
      command: python agent.py --prompt {PROMPT} --out {OUTPUT_FILE}

  - id: azure
    label: azure-grader
    runtime: host
    config:
      endpoint: "{{ env.AZURE_OPENAI_ENDPOINT }}"
      api_key: "{{ env.AZURE_OPENAI_API_KEY }}"
      model: "{{ env.AZURE_DEPLOYMENT_NAME }}"

defaults:
  provider: my-agent
  grader: azure-grader

Your agent.py reads the prompt, writes its response to the path passed as --out, and exits 0. That’s it.

Command contract

Before each test case, AgentV renders the command template and spawns it as a shell process. The command has two responsibilities:

1. Read the input via one of the placeholders below.
2. Write the response to {OUTPUT_FILE} — AgentV reads that file, not your stdout.

When the process exits successfully, AgentV parses the contents of {OUTPUT_FILE} and treats it as the target’s response. Non-zero exits, timeouts, and unreadable output files are surfaced as test errors with the underlying stderr/exit code.

Template placeholders

Use these in command; AgentV substitutes them per test case.

Placeholder	What it expands to
{PROMPT}	The test case’s input text, shell-escaped.
{PROMPT_FILE}	Path to a temp file containing the prompt (use this when the input is large enough to blow past shell argv limits).
{OUTPUT_FILE}	Path to a temp file the command must write to. Deleted after the run unless keep_temp_files: true.
{FILES}	Space-separated paths of any input files attached to the test case, formatted via files_format.
{EVAL_ID}	Unique identifier of the current test case — useful for logging or per-case scratch dirs.
{ATTEMPT}	Retry attempt number (0 on the first try).

Output file format

AgentV tries to parse {OUTPUT_FILE} as JSON first. If it parses and contains any of these keys, they’re picked up; if it doesn’t parse, the entire content is treated as the assistant’s message text.

{
  "output": [                    // preferred: full message array
    { "role": "assistant", "content": "..." }
  ],
  "text": "...",                 // fallback: plain assistant text
  "token_usage": { "input": 123, "output": 456, "cached": 0 },
  "cost_usd": 0.0042,
  "duration_ms": 1800
}

For the common case, plain text is fine:

echo "Hello, world!" > {OUTPUT_FILE}

Configuration fields

Field	Type	Required	Default	Description
label	string	recommended	—	Stable AgentV identity used by defaults.provider, CLI --provider, and comparisons.
provider	literal "cli"	yes	—	Selects this provider.
runtime	string or object	no	host	Target runtime placement.
config.command	string	yes	—	Shell command template.
config.timeout_seconds	number	no	—	Kill the process if it runs longer than this.
config.cwd	string	no	eval dir	Working directory. Relative paths resolve against the eval file.
config.files_format	string	no	{path}	How each entry in {FILES} is formatted. Placeholders: {path}, {basename}.
config.verbose	boolean	no	false	Log the rendered command and cwd to stdout. Useful for debugging template substitution.
config.keep_temp_files	boolean	no	false	Preserve {PROMPT_FILE} / {OUTPUT_FILE} after the run — handy while iterating on your command.
config.healthcheck	object	no	—	Pre-run health check (HTTP or command); the eval aborts if it fails.

Throughput and queueing

AgentV invokes a CLI provider once per eval case. Each invocation receives one rendered prompt or prompt file and must write one response to {OUTPUT_FILE}.

If your underlying tool benefits from throughput batching, keep that batching behind the provider boundary. For example, the command can enqueue the case into a long-running service, call a local daemon that batches requests internally, or use the tool’s native queueing API. The eval YAML should still describe a normal per-case CLI provider:

providers:
  - id: cli
    label: my-agent
    runtime: host
    config:
      command: python agent.py --prompt-file {PROMPT_FILE} --out {OUTPUT_FILE}

The important boundary is that AgentV still receives one result per case. Internal queueing, caching, daemon reuse, or request batching belongs in the adapter or CLI implementation, not in eval-runner batch configuration.

Pattern: Oracle validation (sanity-check your grader)

A common question when building a new eval: “if my grader scores my agent poorly, is the agent wrong or is the grader wrong?” The classical testing answer is to run a known-correct reference (“the oracle”) through the same grader — if a perfect answer doesn’t pass, the grader is the bug.

AgentV has no dedicated “oracle” feature because the cli provider already composes into one. Declare a second target that prints your known-good answer into {OUTPUT_FILE}, run the same eval against it, and assert a perfect score:

.agentv/config.yaml

providers:
  - id: my-agent
    provider: cli
    runtime: host
    config:
      command: python agent.py --prompt {PROMPT} --out {OUTPUT_FILE}

  - id: oracle
    provider: cli
    runtime: host
    config:
      command: cp fixtures/{EVAL_ID}.expected.txt {OUTPUT_FILE}

  - id: azure-grader
    provider: azure
    runtime: host
    config:
      endpoint: "{{ env.AZURE_OPENAI_ENDPOINT }}"
      api_key: "{{ env.AZURE_OPENAI_API_KEY }}"
      model: "{{ env.AZURE_DEPLOYMENT_NAME }}"

defaults:
  grader: azure-grader

# While iterating on your grader, run the oracle first.
# If it doesn't score 100%, fix the grader before trusting any agent results.
agentv eval my.EVAL.yaml --provider oracle

# Then run the real target.
agentv eval my.EVAL.yaml --provider my-agent

A few practical notes:

• {EVAL_ID} in the oracle command lets one target serve an entire eval suite — just ship one fixtures/<id>.expected.txt per case. Alternatively, read the expected output from wherever your rubric already keeps it.
• If the oracle doesn’t reach 100%, that’s the bug. Do not proceed to scoring real agents until it does.
• If the oracle does reach 100%, low scores on real agents are a signal about the agent, not the grader.
• The same composition works for other meta-tests: a “deliberately wrong” target that should score 0, a “mostly right” target pinned at a known partial score, etc.

The pattern needs no special config field, no directory convention, and no flag — it’s just a second target that happens to know the answer.

Use this pattern instead of eval mock dry-run for grader validation. Mock execution was removed because fake candidate answers produced misleading quality failures; a reference target gives deterministic output while exercising the real grader path.

Debugging

When a cli target misbehaves:

1. Set verbose: true to see the rendered command and cwd.
2. Set keep_temp_files: true and inspect {PROMPT_FILE} / {OUTPUT_FILE} after the run.
3. Run the rendered command by hand with those files and check it exits 0 and writes the expected output shape.
4. If the output looks right but grading is off, check the JSON schema — a typo in output vs output_messages silently falls back to “treat whole file as plain text.”