A benchmark that isolates shell quoting/escaping reliability in LLM agents: 14 scenarios × 4 hostility tiers = 56 execution-verified tasks, each solvable by a single bash -c command and scored only by final program state (file bytes, argv a probe receives, git log, parsed JSON) — never by matching the model's command string. Two studies share the task core:

• Study A — controlled stress test (14 models): text transports raw / manual-json / nested-shell. The nested-shell contract interpolates the reply into bash -c "R" — a deliberately adversarial parsing layer that models nested bash -c/ssh wrappers and string-interpolating integrations, not the default transport of mainstream coding agents. It exposes capability-gated failures that raw generation hides.
• Study B — deployment-facing evaluation (5 frontier models, 7,056 task–trial records): raw vs native, where native elicits the command through the serving endpoint's structured shell tool (the command-string boundary Codex CLI, Claude Code's Bash tool, and Gemini CLI share), with full reasoning-effort ladders, 3 trials per cell, and every command executed on BSD and replayed verbatim on GNU.

Headline finding: the structured shell-tool boundary is not a neutral serialization channel. Its effect ranges from a stable benefit (Opus-4.8: +3.1 BSD / +2.6 GNU) to a stable tax (Gemini-3.5-flash: −4.0 / −10.0), with near-null (GPT-5.6-sol) and dialect-dependent sign flips (Gemini-3.1-pro, Fable-5) in between — while tool-call adherence stays at 98.2–100%. Schema-valid invocation does not imply executable correctness.

* = cluster-bootstrap CI excludes zero at both the task level (56 clusters) and the scenario level (14 families). † = task level only (directional). Aggregate over all 7,056 records: +0.3 (BSD) / −2.8 (GNU) — a mean that would read as "no effect" while hiding opposite-signed per-model effects.

pass@1 on the 56-task core, GNU/Linux reference toolchain; sorted by nested-shell. Updated 2026-07-24.

nested-shell interpolates the reply into bash -c "R", adding one parsing layer. It is a controlled adversarial stressor, not the default transport of mainstream coding agents. A large drop (e.g. Gemini-3.5-flash 96.4->67.9) means a model that aces one-shot shell questions cannot survive an extra parsing layer.

aggregate over the effort ladder, 3 trials, GNU replay. native = the structured tool-call API real agents use. * = effect robust at both task- and scenario-cluster level, † = task level only. Updated 2026-07-24.

The tool boundary is not neutral: Opus-4.8 gains, Gemini-3.5-flash loses, and tool-call adherence stays 98-100% throughout, so schema-valid invocation does not imply executable correctness.

This file is a map; facts live in docs/:

• design & metrics → docs/SPEC.md
• novelty & prior art → docs/relatedwork.md
• dataset card → docs/paper/datasheet.md
• license → LICENSE
• citation metadata → CITATION.cff

The public repository and project site are synchronized from the private release source by GitHub Actions.

QuoteBench exposes failures that are easy to miss in broad coding benchmarks:

• The tool interface is a model-visible intervention: it changes which command, utility, and dialect the model emits. On sed-replace, Gemini-3.1-pro's native tool calls pass 38.9% vs raw's 2.8% on BSD — and 55.6% vs 100% when the same commands are replayed on GNU. The measured benefit of a tool boundary can reverse across execution dialects.
• The interface effect interacts with reasoning effort: Opus-4.8's native benefit is largest at low (+7.1 BSD / +8.9 GNU) and vanishes near saturation; Gemini-3.5-flash's native tax grows with effort (−14.3 GNU at high). More reasoning does not immunize a model against an incompatible tool profile.
• Some models are strong under direct shell execution but collapse when the same command is transported through an extra parsing layer (Gemini-3.5-flash 96.4% raw → 67.9% nested-shell, GNU).
• Reasoning budget is not a universal fix: saturated models spend more tokens without improving accuracy; effort labels are vendor-calibrated aliases for a budget, not comparable units.
• pass@1 overstates reliability for tail-risk skills. Every Study B cell runs 3 trials; the interface shifts repeatable competence in both directions (pass³ tasks: Opus-4.8 +15, Gemini-3.5-flash −36, GNU).

python3 -m quotebench list                 # enumerate 56 frozen-core tasks
python3 -m quotebench show ssh-heredoc/t3-gnarly
python3 -m quotebench validate             # oracle 56/56 + discrimination proof
python3 -m quotebench run --adapter naive --out results/naive.jsonl
OPENAI_BASE_URL=http://localhost:8000/v1 OPENAI_API_KEY=... \
    python3 -m quotebench run --adapter openai --model qwen3.5-9b \
    --contract nested-shell --executor docker --out results/qwen.jsonl
OPENAI_BASE_URL=... OPENAI_API_KEY=... \
    python3 -m quotebench run --adapter openai --model MODEL \
    --contract native --executor docker --out results/native.jsonl
python3 -m quotebench score results/qwen.jsonl

Contract names: raw / manual-json / nested-shell / native (legacy spellings json/wrapped remain accepted aliases). No dependencies for the core benchmark (Python 3 stdlib). All benchmark jobs are python3 -m quotebench subcommands (see quotebench/cli.py). For untrusted model output use --executor docker (build with docker build -t quotebench-runner .).

QuoteBench is released under the Apache License 2.0. Redistribution must retain the copyright, license, and NOTICE attribution. If you use QuoteBench in research, reporting, evaluation, derivative benchmarks, or public comparisons, please cite the repository metadata in CITATION.cff.