Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions .changeset/max-tool-calls-middleware.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
'@tanstack/ai': minor
---

Rework tool-call fan-out budgets as middleware hooks (unreleased #965 API).

- **Remove** (never released): `maxToolCalls()` strategy and `chat({ maxToolCallsPerTurn })`
- **Add** `onShouldContinue` middleware hook so policies can stop further agent turns without aborting
- **Keep** `AgentLoopState.toolCallCount` / `lastTurnToolCallCount` for strategies and middleware
- Tool-call budgets are an **app-owned middleware recipe** (docs), not a built-in export

```ts
import { chat, maxIterations, type ChatMiddleware } from '@tanstack/ai'

function toolCallBudget({
max,
maxPerTurn,
}: {
max?: number
maxPerTurn?: number
}): ChatMiddleware {
let perTurn = 0
return {
onIteration: () => {
perTurn = 0
},
onToolPhaseComplete: () => {
perTurn = 0
},
onBeforeToolCall: () => {
if (maxPerTurn == null) return
if (++perTurn > maxPerTurn) {
return {
type: 'skip',
result: {
error: `Skipped: exceeded maxToolCallsPerTurn (${maxPerTurn})`,
},
}
}
},
onShouldContinue: (_ctx, state) =>
max != null && state.toolCallCount >= max ? false : undefined,
}
}

chat({
adapter,
messages,
tools,
agentLoopStrategy: maxIterations(20),
middleware: [toolCallBudget({ maxPerTurn: 10, max: 20 })],
})
```
2 changes: 2 additions & 0 deletions docs/advanced/built-in-middleware.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ TanStack AI ships ready-made middleware so you don't have to hand-roll the commo

> `toolCacheMiddleware` and `contentGuardMiddleware` are exported from the main `@tanstack/ai/middlewares` barrel. `otelMiddleware` lives on its own subpath (`@tanstack/ai/middlewares/otel`) so that importing the barrel never eagerly pulls in `@opentelemetry/api` (an optional peer dependency).

For app-owned policies (for example tool-call budgets), see the [tool-call budget recipe](../chat/agentic-cycle#tool-call-budgets-middleware-recipe) — those stay in your code, not in `@tanstack/ai/middlewares`.

## toolCacheMiddleware

Caches tool call results based on tool name and arguments. When a tool is called with the same name and arguments as a previous call, the cached result is returned immediately without re-executing the tool.
Expand Down
21 changes: 21 additions & 0 deletions docs/advanced/middleware.md
Original file line number Diff line number Diff line change
Expand Up @@ -312,6 +312,26 @@ const redactStructuredOutput: ChatMiddleware = {

> Why is there `onStructuredOutputConfig` but no `onStructuredOutputChunk`? Because the **config** shape genuinely differs at the structured-output boundary — it carries an `outputSchema` field that plain `ChatMiddlewareConfig` doesn't (see [onStructuredOutputConfig](#onstructuredoutputconfig)). **Chunks** are all just `StreamChunk` regardless of phase, so one `onChunk` plus `ctx.phase` (or the CUSTOM event name) covers every case — a parallel chunk hook would be redundant.

### onShouldContinue

Called when the engine is deciding whether to start another agent-loop iteration (after a tool phase or between model turns). Combined with AND semantics across middleware **and** with `agentLoopStrategy` — any explicit `false` stops the loop. Return `true`, `void`, or `undefined` to allow continuation.

Does **not** abort the run: the stream finishes normally with the current messages. Use `ctx.abort()` only for a hard abort.

```typescript
import { type ChatMiddleware } from "@tanstack/ai";

const budget: ChatMiddleware = {
name: "tool-budget",
onShouldContinue: (_ctx, state) => {
// Stop further turns once 20 tool calls have been emitted
if (state.toolCallCount >= 20) return false;
},
};
```

For a full per-turn + cumulative tool budget recipe, see [Tool-call budgets](../chat/agentic-cycle#tool-call-budgets-middleware-recipe).

### onBeforeToolCall

Called before each tool executes. The first middleware that returns a non-void decision short-circuits — remaining middleware are skipped for that tool call.
Expand Down Expand Up @@ -596,6 +616,7 @@ const stream = chat({
| `onStart` | Sequential | All run in order |
| `onChunk` | **Piped** — chunks flow through each middleware | If first drops a chunk, later middleware never see it |
| `onBeforeToolCall` | **First-win** — first non-void decision wins | Earlier middleware has priority |
| `onShouldContinue` | **AND** — any explicit `false` stops the loop | Order only affects which middleware runs first when short-circuiting |
| `onAfterToolCall` | Sequential | All run in order |
| `onUsage` | Sequential | All run in order |
| `onFinish/onAbort/onError` | Sequential | All run in order |
Expand Down
35 changes: 3 additions & 32 deletions docs/api/ai.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ const stream = chat({
- `tools?` - Array of tools for function calling
- `context?` - Typed runtime context passed to server tools and middleware. If a tool or middleware declares a concrete context type, `chat()` requires a compatible value here
- `systemPrompts?` - System prompts to prepend to messages
- `agentLoopStrategy?` - Strategy for agent loops (default: `maxIterations(5)`). Strategies receive `{ iterationCount, finishReason, messages, toolCallCount, lastTurnToolCallCount }` and run between model turns. Prefer `maxToolCalls(n)` when you need a tool-call budget — iterations are model turns, not tool calls. `maxToolCalls` stops *further* turns once cumulative **emitted** tool calls are `>= n`; the turn that crosses the limit is not truncated.
- `maxToolCallsPerTurn?` - Cap how many tool calls from a single model turn (or pending/resume batch) are executed. Excess calls receive error results so message history stays consistent. Unset means no per-turn execution cap; `0` skips all execution for that batch. Pair with `maxToolCalls(n)` for a cumulative **emitted**-call budget (skipped calls still count).
- `agentLoopStrategy?` - Strategy for agent loops (default: `maxIterations(5)`). Strategies receive `{ iterationCount, finishReason, messages, toolCallCount, lastTurnToolCallCount }` and run between model turns. Iterations are model turns, not tool calls — for tool-call budgets use middleware (`onBeforeToolCall` + `onShouldContinue`); see [Tool-call budgets](../chat/agentic-cycle#tool-call-budgets-middleware-recipe).
- `middleware?` - Array of chat middleware. Use `onShouldContinue` / `onBeforeToolCall` for app-owned tool budgets.
- `abortController?` - AbortController for cancellation
- `modelOptions?` - Provider-native model options. This is where sampling parameters live — `temperature`, `top_p`/`topP`, and the provider's token-limit key (`max_output_tokens`, `max_tokens`, `maxOutputTokens`, …) — under each provider's canonical name, rather than as generic root-level props. See [Moving Sampling Options into modelOptions](../migration/sampling-options-to-model-options). (Renamed from `providerOptions`.)
- `threadId?` - AG-UI thread identifier propagated into `RUN_STARTED` events for run correlation
Expand Down Expand Up @@ -315,7 +315,7 @@ A merged tool record suitable for `chat({ tools })`.

## `maxIterations(count)`

Creates an agent loop strategy that limits **model turns** (iterations), not tool calls. One turn can still emit many parallel tool calls — use `maxToolCalls` and/or `maxToolCallsPerTurn` for tool-call budgets.
Creates an agent loop strategy that limits **model turns** (iterations), not tool calls. One turn can still emit many parallel tool calls — use middleware for tool-call budgets ([recipe](../chat/agentic-cycle#tool-call-budgets-middleware-recipe)).

```typescript
import { chat, maxIterations } from "@tanstack/ai";
Expand All @@ -336,35 +336,6 @@ const stream = chat({

An `AgentLoopStrategy` function.

## `maxToolCalls(count)`

Creates an agent loop strategy that continues while cumulative **emitted** tool calls are below `count` (including calls skipped by `maxToolCallsPerTurn`).

Strategies only run between turns — the turn that crosses the limit is not truncated, so the final count may exceed `count` unless you also set `maxToolCallsPerTurn`. Pair with `chat({ maxToolCallsPerTurn })` to cap parallel fan-out inside a single turn.

```typescript
import { chat, combineStrategies, maxIterations, maxToolCalls } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";

const stream = chat({
adapter: openaiText("gpt-5.2"),
messages: [{ role: "user", content: "Hello!" }],
maxToolCallsPerTurn: 10,
agentLoopStrategy: combineStrategies([
maxIterations(20),
maxToolCalls(20),
]),
});
```

### Parameters

- `count` - Maximum cumulative tool calls across the run

### Returns

An `AgentLoopStrategy` function.

## Types

### `ModelMessage`
Expand Down
90 changes: 68 additions & 22 deletions docs/chat/agentic-cycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,17 +159,17 @@ The loop continues only while the model's finish reason is `tool_calls` (with pe

By default the loop is bounded by `maxIterations(5)` — after five **model turns** it stops even if the model would keep calling tools. Override this with the `agentLoopStrategy` option.

> **Iterations ≠ tool calls.** One model turn can emit many parallel tool calls. `maxIterations` only bounds turns. Use `maxToolCalls(n)` for a cumulative **emitted**-call budget (stops further turns once the count is reached; the crossing turn is not truncated), and `maxToolCallsPerTurn` to cap how many of those parallel calls **execute** in a single turn or pending/resume batch (strategies only run *between* turns, so without a per-turn cap a single runaway turn can still fan out unbounded). Skipped calls still count toward `maxToolCalls`.
Other built-in strategies:

- **`untilFinishReason([...])`** — continue until the model returns one of the given finish reasons (e.g. `untilFinishReason(["stop", "length"])`).
- **`combineStrategies([...])`** — combine multiple strategies with AND logic; the loop continues only while every strategy agrees.

A strategy is just a function that receives `{ iterationCount, finishReason, messages, toolCallCount, lastTurnToolCallCount }` and returns `true` to allow another iteration or `false` to stop, so you can also write your own:

```typescript
import {
chat,
combineStrategies,
maxIterations,
maxToolCalls,
toServerSentEventsResponse,
} from "@tanstack/ai";
import { chat, combineStrategies, maxIterations, toServerSentEventsResponse } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import type { AgentLoopState } from "@tanstack/ai";
import { getWeather, getClothingAdvice } from "./tools";

export async function POST(request: Request) {
Expand All @@ -178,41 +178,87 @@ export async function POST(request: Request) {
adapter: openaiText("gpt-5.5"),
messages,
tools: [getWeather, getClothingAdvice],
maxToolCallsPerTurn: 10, // cap parallel fan-out inside one turn
agentLoopStrategy: combineStrategies([
maxIterations(20), // max model turns
maxToolCalls(20), // max tool calls across the run
maxIterations(10),
({ messages }: AgentLoopState) => messages.length < 100,
]),
});
return toServerSentEventsResponse(stream);
}
```

Other built-in strategies:
### Tool-call budgets (middleware recipe)

- **`maxToolCalls(n)`** — continue while cumulative emitted tool calls are below `n` (including per-turn skips; not model turns).
- **`untilFinishReason([...])`** — continue until the model returns one of the given finish reasons (e.g. `untilFinishReason(["stop", "length"])`).
- **`combineStrategies([...])`** — combine multiple strategies with AND logic; the loop continues only while every strategy agrees.
> **Iterations ≠ tool calls.** One model turn can emit many parallel tool calls. `maxIterations` only bounds **model turns**. Strategies run *between* turns, so without a per-turn cap a single runaway turn can still fan out unbounded.

A strategy is just a function that receives `{ iterationCount, finishReason, messages, toolCallCount, lastTurnToolCallCount }` and returns `true` to allow another iteration or `false` to stop, so you can also write your own:
There is no built-in `maxToolCalls` strategy. Cap tools with middleware:

- **`onBeforeToolCall`** — skip excess calls inside one turn (`maxPerTurn`)
- **`onShouldContinue`** — stop further turns once cumulative **emitted** tools hit a budget (`max`); skipped calls still count toward `toolCallCount`

```typescript
import { chat, combineStrategies, maxIterations, toServerSentEventsResponse } from "@tanstack/ai";
import {
chat,
maxIterations,
toServerSentEventsResponse,
type ChatMiddleware,
} from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import type { AgentLoopState } from "@tanstack/ai";
import { getWeather, getClothingAdvice } from "./tools";

/** App-owned policy — not a library export. */
function toolCallBudget(options: {
max?: number;
maxPerTurn?: number;
}): ChatMiddleware {
const { max, maxPerTurn } = options;
let perTurn = 0;

return {
name: "tool-call-budget",
onIteration() {
perTurn = 0;
},
// Fresh per-turn budget for pending/resume batches (no onIteration).
onToolPhaseComplete() {
perTurn = 0;
},
onBeforeToolCall() {
if (maxPerTurn == null) return undefined;
perTurn += 1;
if (perTurn > maxPerTurn) {
return {
type: "skip",
result: {
error: `Skipped: exceeded maxToolCallsPerTurn (${maxPerTurn})`,
},
};
}
return undefined;
},
onShouldContinue(_ctx, state) {
if (max != null && state.toolCallCount >= max) return false;
return undefined;
},
};
}

export async function POST(request: Request) {
const { messages } = await request.json();
const stream = chat({
adapter: openaiText("gpt-5.5"),
messages,
tools: [getWeather, getClothingAdvice],
agentLoopStrategy: combineStrategies([
maxIterations(10),
({ messages }: AgentLoopState) => messages.length < 100,
]),
agentLoopStrategy: maxIterations(20), // model turns
middleware: [
toolCallBudget({
maxPerTurn: 10, // cap parallel fan-out inside one turn
max: 20, // stop further turns once cumulative emitted tools hit 20
}),
],
});
return toServerSentEventsResponse(stream);
}
```

Place this **before** `toolCacheMiddleware` so over-budget skips win over cache hits. See [`onShouldContinue`](../advanced/middleware#onshouldcontinue) for the hook contract.
9 changes: 5 additions & 4 deletions docs/config.json
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@
"label": "Agentic Cycle",
"to": "chat/agentic-cycle",
"addedAt": "2026-04-15",
"updatedAt": "2026-07-20"
"updatedAt": "2026-07-21"
},
{
"label": "Streaming",
Expand Down Expand Up @@ -337,12 +337,13 @@
"label": "Middleware",
"to": "advanced/middleware",
"addedAt": "2026-04-15",
"updatedAt": "2026-06-15"
"updatedAt": "2026-07-21"
},
{
"label": "Built-in Middleware",
"to": "advanced/built-in-middleware",
"addedAt": "2026-06-03"
"addedAt": "2026-06-03",
"updatedAt": "2026-07-21"
},
{
"label": "OpenTelemetry",
Expand Down Expand Up @@ -502,7 +503,7 @@
"label": "@tanstack/ai",
"to": "api/ai",
"addedAt": "2026-04-15",
"updatedAt": "2026-07-20"
"updatedAt": "2026-07-21"
},
{
"label": "@tanstack/ai-client",
Expand Down
1 change: 0 additions & 1 deletion docs/reference/functions/combineStrategies.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ const stream = chat({
tools: [weatherTool],
agentLoopStrategy: combineStrategies([
maxIterations(10),
maxToolCalls(20),
({ messages }) => messages.length < 100,
]),
});
Expand Down
6 changes: 3 additions & 3 deletions docs/reference/functions/maxIterations.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ Defined in: [packages/ai/src/activities/chat/agent-loop-strategies.ts:25](https:
Creates a strategy that continues for a maximum number of **model turns**
(iterations), not tool calls.

One iteration can still emit many parallel tool calls. Prefer
[maxToolCalls](maxToolCalls.md) (and optionally `maxToolCallsPerTurn` on `chat()`)
when you need a tool-call budget.
One iteration can still emit many parallel tool calls. Prefer middleware
with `onBeforeToolCall` / `onShouldContinue` when you need a tool-call
budget (see the Agentic Cycle docs recipe).

## Parameters

Expand Down
53 changes: 0 additions & 53 deletions docs/reference/functions/maxToolCalls.md

This file was deleted.

1 change: 0 additions & 1 deletion docs/reference/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,7 +294,6 @@ title: "@tanstack/ai"
- [isProviderExecutedToolCall](functions/isProviderExecutedToolCall.md)
- [isStandardSchema](functions/isStandardSchema.md)
- [maxIterations](functions/maxIterations.md)
- [maxToolCalls](functions/maxToolCalls.md)
- [mergeAgentTools](functions/mergeAgentTools.md)
- [modelMessagesToUIMessages](functions/modelMessagesToUIMessages.md)
- [modelMessageToUIMessage](functions/modelMessageToUIMessage.md)
Expand Down
Loading
Loading