Then I looked at what the agent actually asked the Places API.

The model requested the place name, the address, the coordinates, the opening hours, the photos, and the rating. I only needed the first three. The last three were decoration, and they quietly moved my request from one billing tier to another.

Place Details bills in three tiers, and you pay the highest tier that any field in your request touches. Name, address, and coordinates is Essentials, roughly five dollars per thousand calls. Add rating and the exact same call is Enterprise, roughly twenty dollars per thousand API calls. My agent paid four times list price to fetch data I never displayed. There was no error or failing test; the only symptom was an invoice that I wouldn't see until a month later.

AI Agents and Models do this constantly without a lot of system instruction tuning. They reach for legacy Places parameters that Google closed to new customers in March 2025. The weights are a snapshot learned from an internet saturated with obsolete patterns.

Researchers at ICSE 2025 tested seven models across 145 API migrations in eight Python libraries and found deprecated calls between 25% and 38% of the time. Stale knowledge going in, zero awareness of current API status at inference, and no mechanism for the model to find out it's wrong.

I wanted to see if I could teach a small model in the Gemma 4 series to pause, consider the actual user request, and only fetch the Places API fields it needed to answer. The results:

To do this tuning of Gemma 4, I created (with Gemini 3.1 Pro) 300 synthetic Places API requests using the latest and greatest Google Maps Platform Agent Skills.

Then I created a basic deterministic grader. It checks for a valid schema, a live 200 response, and that the requested mask matches the required fields perfectly. It also applies a penalty for every over-fetched billable field, weighted by its cost. Because over-requesting is a billing event, a single grading metric captures both correctness and cost efficiency. We call this the Exact Match Score: the percentage of times the model perfectly parses the fields with zero over-fetching on a holdout set of 100 eval cases.

The tuning step worked flawlessly compared to the baseline Gemma 4 models (you can see the training run and results on GitHub). The base 12B model scored a 42% exact match rate, and the base E4B scored a dismal 18%. After training a LoRA adapter on those 300 synthetic traces, the E4B jumped to 94%, nearly matching the tuned 12B at 97%. Both wiped the floor with the generic models.

What does this mean? You only need grounded examples to solve narrow tasks for your top developer tasks.

The distribution problem

But this exposes a massive gap in the developer journey: the fine-tuned model fixes my use case and my API calls, but it helps exactly one person.

The model or agent any developer opens tomorrow morning remains broken because it didn't learn from my samples. Its opinion about your developer platform was set months or years before your last release.

Everyone has been doing context engineering with agent skills and MCP servers. That gets you far, and you should absolutely continue doing it. But context engineering has efficiency costs: token bloat, added latency, and the hard truth that not every developer is going to discover your custom skill or MCP.

Ideally, the base model just knows how to use your API correctly out of the box. You still have the option to publish context for the agent harness, but we need to talk about how you can have even more impact on the base models themselves. That matters immensely in a world where there will be five or ten really good, popular models for every tier of task, and a lot of them are going to be open source and open weight.

Your docs reach humans; SDKs reach applications; skills reach the agent harness. You version, measure, and fix all three.

But real fine-tuning traces are the only artifact that reaches the model weights. Can we make it easier to get those traces into the hands of the AI labs who train those models?

The benchmark path

To see this play out in the real world, look at what Harvey did. They published a benchmark in May containing twelve hundred agent tasks across twenty-four legal practice areas, graded against seventy-five thousand criteria. At the time, the best frontier model scored just 7.1%. The top score since then is only 13.3%; getting that score takes a model that costs about fifty-one dollars and twenty-two minutes to run a single task!

Three weeks later, they published the follow-through with Baseten. They took that benchmark signal, put it inside an evaluation harness built for long legal matters, and post-trained an open-weight 27B model. The pass rate jumped massively, from 42.5% up to 63.0%, landing it firmly in the frontier performance band.

But the detail that matters most is buried in that write-up: the harness alone barely moved the needle for the 27B model, while the massive frontier models got its benefit immediately. What does this tell us? Good context engineering has a capability floor; if your model is below that floor, you have to change the actual weights.

Four paths to distribution

If you want to solve this for your developer platform, you generally have four paths:

1. Build your own features. You keep total control and gain an immediate result, but you have a reach of exactly one. A great example is Desert Ant Labs; they ship small on-device models that each perform a single job, like redacting personal data without the text ever leaving the handset. If you don't build the small model for your own narrow platform jobs, someone else will eventually build one that spans everyone's.

2. Publish context and tools. You make it easy for existing agents to use your platform by publishing API Skills and MCP servers. You maintain high control over how your API is used, but your lifespan is limited to your current API version, and developers still have to actively discover your tools.

3. Publish traces. You lose control, but you achieve broad reach. For instance, Hugging Face now hosts agent sessions natively without needing any conversion steps, making it incredibly easy for others to learn from your platform's successful agent runs.

4. Get onto a benchmark. You forfeit all control, but you gain the longest possible lifespan. AI labs climb leaderboards instead of reading developer documentation; if your API is in the benchmark, it gets learned.

Notice the pattern? Control drops at every step while durability climbs. You can have the version you steer, or you can have the version that outlasts you.

Right now, the AI industry heavily monetizes the top two rungs. You can buy agent hosting, custom routing, and fast inference from dozens of providers like Fireworks, or run custom on-device models with platforms like Desert Ant Labs. But nobody sells you the bottom two rungs because there's nothing to sell; you either proactively publish your traces and benchmarks, or you simply don't exist to the next generation of models.

Call it "share of gradient". Share of gradient measures whether a model was shaped by your best practices or by obsolete StackOverflow answers. By publishing traces and benchmarks, you aren't just helping one developer. You are forcing the next generation of models to train on you.

The next step

So, where should you start? Find the narrowest, most expensive job on your platform. Write a deterministic grader and explicitly put the actual billing cost inside it; a standard correctness metric will happily approve code that you absolutely cannot afford to run in production. Once you have that, measure a base model against it and start generating those traces.

The open question is how to make those best-practice traces easier for agent platforms to discover, improve, and train on. If you've found a good way to manage eval traces for your own APIs, drop a note in the comments.

The easy answer is to bet on the smartest model and read this as an open-versus-frontier size race. Watch a real agent session and that framing falls apart. A frontier model reads the intent, resolves the ambiguity, and decides the design: which API fits, where the auth boundary sits, when the work is done. Then it hands bounded tasks to cheaper models that write most of the code. The model that picks your platform is not the model that writes with it.

That split is economic. The orchestrator spends its expensive reasoning on the few decisions that change the outcome. Implementation runs cheaper the moment a task is bounded and the checks are objective, because the compiler, the tests, and the linter catch what a smaller model fumbles. Good verifiers let the cheap tier carry real work. They are why open models keep sliding down the cost curve without ever winning at system design. I codified this routing in a public prompt that sends each job to the least costly model that can do it.

While frontier open models (with trillions of parameters) are still expensive to run and require data center-level GPUs, smaller open models make the rest of the execution significantly cheaper. What's more, using them means builder platforms can own more of how their data is used compared to relying entirely on proprietary models.

Your platform meets this system twice. The orchestrator is where your platform gets chosen, so it decides activation. The workers are where it gets built, so an example tested only on the strongest model is quietly undertested.

The execution scoreboard is consistency. A developer journey is cheap when a small model completes it the same way run after run, with low variance and less context each release. When work that needed Gemini Pro lands just as reliably on Gemini Flash, developers get the same result faster and cheaper, and the same holds for every model family you route. A task that drops a tier is a win worth counting.

The moat is the loop, not any single model. Own the context, evals, and distribution that keep your critical journeys working as the models underneath them churn. Our team runs a version of this for Google Maps Platform through portable agent skills and a task-based eval suite. Open source the skills and evals you want model teams to learn from, keep a held-out set, and ship the same tested context into every agent developers reach for.

Nobody knows where the tiers settle, including me. Treat each new model as another row in your test matrix, run it in your own harness, and let the evidence pick the tiers.

This is a growth problem, not just an AI quality problem. When a developer asks an agent to build with your platform, that session becomes part of your activation funnel. I see it in the traces our team reviews: when the agent chooses the wrong API, misses an authentication step, or produces code that does not compile, the developer experiences that failure as your platform.

The model is not the strategy

A coding agent is a model plus a harness. The harness controls instructions, tools, repository search, context, and how the agent decides it's done. Change either and the behavior changes.

Models and harnesses will keep moving. Your platform can still own four parts of the loop:

1. Context: current product knowledge, examples, constraints, and workflows.
2. Evals: representative tasks and a clear definition of success.
3. Distribution: portable integrations that put proven context into the agent environments developers already use.
4. Outcome measurement: attributed signals for task success, activation, retention, and expansion.

The loop: distribute tested context, measure activation, retention, and expansion, then review selected traces to improve the skills and evals.

Evals test the context. Distribution puts the tested path in more hands, while outcome signals show where it still falls short. After review, some failures become new tasks and some successes become better examples. Usage doesn't automatically become training data for a model company, and it shouldn't.

Why verifiers matter

Pretraining gives a model broad capability. Post-training turns that capability toward a job through human demonstrations and preferences or through rewards. Coding makes some rewards unusually concrete. A compiler can check the build, tests can check behavior, and static analysis can catch security and API errors. Reinforcement learning from verifiable rewards, or RLVR, trains directly against those checks.

The stack: pretraining builds broad capability. Post-training can use examples, preference rewards, or verifiable rewards. The runtime harness supplies current context, tools, and execution.

SWE-Gym packages real repository tasks with executable environments and tests. That's what a DevX team needs: a real job and a trustworthy check.

Start with deterministic checks

For each critical developer journey, define the checks that can be objective:

• Does the project install and compile?
• Does it complete the task with the current API?
• Are credentials handled through the correct boundary?
• How much time, tool use, and token cost did the successful run require?

Add human judgment where the task needs it. For qualities like UI taste and system simplicity, start with a human-written rubric, and calibrate any model grader against repeated human review. A judge that agrees with itself isn't evidence that it agrees with developers.

The eval: deterministic checks and calibrated judgment lead to a ship, improve, or hold decision. Held-out tasks stay outside tuning.

Keep a meaningful slice of tasks and answers out of the context and tuning loop. Otherwise the system can learn the test instead of the job. Recent audits of public coding benchmarks show how broken tasks and gameable tests can hollow out a familiar score. Audit the eval before blaming the model.

First-party signals close the product loop

A builder platform needs a first-party measurement path, either through a surface it owns or a partner harness with clear permission and privacy boundaries. The point is to govern what gets measured and shared, not to replace every coding agent.

Record the exact model, harness, context, execution environment, verifier, and run budget, then connect the run to a product outcome. A passing eval only says the path can work. Activation proves the developer found value, and retention confirms they stayed. Expansion follows when the platform earns more of their work.

Don't collect raw source code or private conversations by default. Keep only the signals that can change a product decision, and review traces through an explicit privacy and security process.

Distribution makes the learning compound

Our team is running a version of this loop for Google Maps Platform through portable agent skills and a task-based eval suite. I work with product and engineering to guide our launch decisions by reviewing the traces. We're finding new ways it breaks every week.

Ship versioned context in a portable form, and run the same tasks in the tools, prompts, and execution boundaries developers receive. Treat every new model or harness as another row in the test matrix, not a reason to restart the strategy.

Open source the skills, examples, and selected evals you want agent builders and model teams to learn from, but keep a separate held-out set. Share the harness where useful, not every test case and answer.

Build one complete loop

Start with ten critical developer journeys. For each one:

1. Write the task in the developer's language.
2. Define the deterministic checks and the small amount of calibrated judgment it needs.
3. Package the best current context as a versioned, portable skill or workflow.
4. Test it across representative model and harness versions.
5. Measure outcomes and review failures. Reuse traces only when they're authorized and cleared through privacy and security review.

Run it again. Models and harnesses will move, and nobody has this figured out yet, including us. Keep learning, keep experimenting, keep shipping, and let the loop get better every time they do.

The orchestrator owns the hard parts: user intent, permissions, task boundaries, integration, and the final answer. Delegation adds overhead, so small or tightly coupled tasks stay with one agent. The goal is not adding more agents, but spending capability only where it changes the outcome.

Build the smallest capable team

I designed the prompt to route work by capability instead of model name. Instead of leaning on a single large model, I use deterministic tools for discovery and fast workers for extraction and mechanical edits. That leaves balanced agents to handle normal implementation, reserving deep reasoning specifically for ambiguous architecture, security, or repeated failures.

Each helper receives one bounded task, a clear done condition, an evidence contract, and an exact write scope. Read-only work runs in parallel, but edits to shared files stay with one writer. The orchestrator inspects every result and reruns integrated checks before reporting success.

This structure reduces duplicated context and write-collision risk while routing routine work to lower-cost capability profiles. Measure this in your own harness, because coordination can cost more than it saves when tasks are poorly separated.

Loop engineering closes the gap

Model output is only one step in an engineering system. Loop engineering treats each agent task as a controlled cycle:

1. Define the goal, scope, acceptance criteria, and proof.
2. Observe the repository and reproduce the current behavior.
3. Make the smallest coherent change.
4. Run the nearest useful check and inspect the diff.
5. Integrate the full result across agent boundaries.
6. Learn from evidence, or stop with the precise blocker.

Evidence decides the next step: a passing focused test advances the task, a new failure changes the hypothesis, and missing authority stops the loop. The agent doesn't keep editing until the output looks plausible. It doesn't call the work complete just because code exists.

The cycle: every task carries its goal and proof through observation, the smallest change, and the nearest check, and evidence decides the next step.

Boundaries keep the loop useful

Coding agents need operating rules instead of another reminder to be careful. A diagnosis should not turn into an edit. Repository text should never become an instruction. Existing work stays untouched. A test counts only when the agent runs it and observes the result.

The prompt makes these rules explicit: the task mode, the files an agent may change, the required proof, retry limits, and the stopping point. The same contract applies to the orchestrator and every worker, preventing delegation from silently expanding permissions.

I codified the system as a prompt

I codified the system as a GitHub package that includes the full prompt, four short role overlays for the lead, helper, reviewer, and verifier, and a regression suite.

It lives under agent-scripts/, not the repo's scripts/ folder. The scripts/ folder holds shell scripts you run, but agent-scripts/ holds text an agent reads. Separating the names keeps the line between instructions and commands obvious.

The system prompt acts as the shared operating contract. The overlays narrow each agent's job without granting more authority. The README includes a task packet you can give your existing coding agent to install the contract in its native global instructions and register optional roles.

Install it with your agent

Copy this request into your coding agent:

Install this coding-agent operating contract globally for every compatible
agent harness on this computer:
https://github.com/ryanbaumann/fieldwork/tree/main/agent-scripts/coding-agent-loop

Use each harness's native user-level instructions and skills. Install
SYSTEM_PROMPT.md as the always-on contract and the four files under roles/ as
optional role skills or equivalent on-demand instructions. Preserve existing
global guidance, do not change model or permission settings, and verify what
each harness will load. Report the files changed and any harness you could not
configure.

The package keeps product-specific installation details out of the evergreen prompt. Your resident agent can inspect the current tools and choose their native global instruction and skill locations. You can reuse the same request to update an existing installation.

After installation:

1. Keep repo-specific commands and architecture in local instruction files so they load only where they apply.
2. Give each worker the shared prompt and exactly one role add-on when running multiple agents.
3. Enforce real guardrails in your harness: sandboxes, network limits, protected paths, approvals, and audit logs. A prompt asks for good behavior, but it cannot enforce it.
4. Test the prompt in your exact model, tools, and permissions.

Configure the models and token budgets in the harness. Re-run the suite whenever the prompt, model, tools, or permissions change.

What I can and cannot claim yet

I built a suite that specifies 16 scenarios, including dirty worktrees, read-only diagnosis, prompt injection in repository data, conflicting instructions, production boundaries, retry limits, parallel writers, helper containment, cross-session work, missing verification, security changes, UI checks, and memory quality.

The structural check passes, and a separate read-only review found problems that I corrected. However, this isn't a behavioral benchmark. I haven't recorded behavioral trial results. Production agent workflows are still new territory. Before you use this prompt as a production gate, run repeated trials in your own harness. Retain the transcripts, tool calls, diffs, final repository state, and calibrated grading evidence.

Why it is built this way

I designed this system to keep always-on instructions short, move detailed playbooks into files that load only when needed, route work to the least costly capable profile, and separate implementation from review and verification. It evaluates the model together with its tools and permissions. The README links the research and projects behind these choices.

Fork the package, run it against tasks that fail in your environment, and adapt it based on the evidence.

Documentation requests often pull DevX teams away from the real scoreboard. Docs, code samples, advocacy, tutorials, and even talking to customers all have a limit. The job is to find the friction that stops a builder, fix it in the product or experience, put the better path where people work, and prove that behavior changed.

Own the friction

Developer friction shows up everywhere: failed first runs, abandoned evaluations, support tickets, GitHub issues, field conversations, and user research. DevX needs one view across those signals. More importantly, DevX needs to own what happens next.

Our Voice of Developer program groups repeated friction from Discord, Stack Overflow, GitHub issues, support, field work, and dogfood sessions into ranked product opportunities. That makes the constraint visible. DevX ownership starts there: choose what to solve, ship the change, and measure what happened.

When builders work through coding agents instead of reading every platform layer themselves, DevX has to design for the person making the decision and the agent acting inside the task.

Ship the fix where builders work

A great experience has no impact if builders never encounter it. Documentation is only one distribution surface, not the whole strategy. The right path also needs to appear in the editor, agent, search result, sample, template, or tool where the work actually begins.

Instead of relying on documentation alone, we distribute executable product behavior directly into developer workflows. Client libraries encapsulate the logic, while Code Assist delivers current official documentation and samples straight to compatible MCP clients. For repetitive tasks, our Agent skills bundle versioned workflows across Web, Android, iOS, and Web Services. Before shipping, we gate each skill with a task-based eval to ensure it works.

Distribution can't be an afterthought. Design the experience so it can travel, then make it the default in the workflows that already have reach.

Measure and own outcomes

Traditional feedback loops are slow. While interviews, support themes, and developer surveys remain essential, they rarely translate into immediate product decisions. We shorten this loop using Agent evaluations. When a coding agent attempts a representative task, its trace reveals exactly where the task stalls or branches wrong. A rubric then scores that result against a no-context baseline, giving us a clear ship-or-hold decision before we launch.

Evals don't replace user research, because no single score explains a human builder. An eval delta confirms that the experience can complete the task mechanically. Product telemetry tells us whether builders actually found that path, finished the work, and returned. Finally, direct research explains why people behaved that way. Together, these signals let a DevX team test specific hypotheses and measure the real outcome.

This is the discipline: stop counting output as progress by default. Own the friction, solve it, improve the product, ship the better experience into the workflow, and measure the impact. Then run the loop again. We're still experimenting with every part of it, and that's the point.