Runtime reliability for AI agents. Structural and semantic detection, runtime prevention, native root cause, and one-click fixes.

If Dunetrace helps you, consider giving it a ⭐ on top right, it helps others find the project.

AI agents fail silently:

• ✓ API returns 200   ✓ Logs are clean
• ✗ Agent called the same tool 12 times, burned $10, and gave the user a wrong answer

Tracers answer "what happened?" — after you already know it broke. Dunetrace answers "is something breaking right now?" and fires an alert in 15 seconds, using zero-LLM structural checks that run in-path with sub-500μs per-hook overhead.¹

Dunetrace covers the full agent reliability lifecycle, not just one slice of it:

¹ Per instrumentation hook (tool_called, llm_responded, …) — benchmarked in packages/sdk-py/tests/test_benchmark.py. The one exception is the prompt-injection scan at run start, which is bounded rather than sub-millisecond: it scans up to 32K characters of input, ~10ms worst case against an LLM call of 500ms+. See docs/detectors.md footnote 2.

Where tracers fit in: if you already run Langfuse, LangSmith, or Braintrust, Dunetrace pulls their evaluation results in alongside its own (pillar 3) rather than asking you to switch — see docs/integrations/external-evaluation.md. What no tracer does is pillar 4: none of them can stop a run mid-flight, because none of them run in-path.

See the examples index for ready‑to‑run examples.

1. Start the backend

git clone https://github.com/dunetrace/dunetrace
cd dunetrace && cp .env.example .env
docker compose -f docker-compose.ghcr.yml up -d
pip install -r requirements.txt

2. Install the SDK

pip install dunetrace                       # Python
npm install dunetrace                       # Node.js / TypeScript

3. Instrument your agent

Python

from dunetrace import Dunetrace
import openai

dt = Dunetrace()
dt.init(agent_id="support-agent")   # auto-instruments installed clients (OpenAI, Anthropic, LangChain, CrewAI, httpx, requests)

@dt.agent("support-agent", model="gpt-4o")
def my_agent(question: str) -> str:
    resp = openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": question}],
    )
    return resp.choices[0].message.content   # LLM + tool calls tracked automatically, no manual hooks

TypeScript / Node.js

import { Dunetrace, autoInstrument } from "dunetrace";
import OpenAI from "openai";

const dt = new Dunetrace();
autoInstrument({ openai: OpenAI });   // patches OpenAI + outbound fetch; add `anthropic: Anthropic` if you use it

const openai = new OpenAI();          // constructed after the patch — still tracked

await dt.run("support-agent", { model: "gpt-4o" }, async (run) => {
  await openai.chat.completions.create({ model: "gpt-4o", messages });
  run.finalAnswer();                  // LLM + tool calls tracked automatically, streaming included
});

To instrument a single client instead, use dt.wrapOpenAI(new OpenAI()). See docs/integrate-typescript-agent.md.

Try the built-in failure scenarios

Python — run from packages/sdk-py:

cd packages/sdk-py

python examples/basic_agent.py                          # No LLM calls
SCENARIO=tool_loop python examples/langchain_agent.py   # TOOL_LOOP via LangChain
SCENARIO=failures python examples/decorator_agent.py    # TOOL_LOOP, RETRY_STORM, RAG_EMPTY_RETRIEVAL

TypeScript: run from packages/sdk-ts. Drives the Vercel AI SDK against a local Ollama, so no API key is needed:

cd packages/sdk-ts
npm install && ollama pull llama3.2

npm run example:vercel-ai                               # Happy path
npm run example:vercel-ai:loop                          # TOOL_LOOP, then prints the root cause and fix

The :loop variant provokes a tool loop, polls until the detector picks it up, then calls POST /v1/signals/{id}/explain — so it exercises the full detect → explain path end to end. That explain call spends one LLM call on whatever provider the stack is configured with; the agent's own calls are free.

Open the dashboard: http://localhost:3000

29 detectors run on every completed run — no configuration, no LLM. A few of the main ones:

Each alert includes: what fired, why it matters, a concrete fix, and a rate context line (first occurrence / recurring / systemic).

→ docs/detectors.md for the full list of 29 detectors

Multi-agent systems — instrument each agent as its own dt.run() and Dunetrace auto-links them into a delegation graph (parent_run_id is threaded automatically for nested runs). Two detectors read that graph: DELEGATION_LOOP (agents cycling without converging) and HANDOFF_CONTEXT_LOSS (a handoff dropping the parent's context). → docs/multi-agent.md

Agent memory — instrument what an agent writes to and reads from its own memory (run.memory_written() / memory_read(), or automatically for LangGraph/CrewAI memory via dt.auto_instrument()), and MEMORY_POISONING flags adversarial content persisted into it. → docs/memory.md

Custom detectors — write a detector in plain English. Dunetrace translates it to a structured condition set, runs it in shadow mode against real traffic, and lets you review the fire rate before any alert fires. In the dashboard: Config → Custom detectors → Add detector.

Detector packs — opt-in detector bundles for a specific class of agent, activated per org. The voice pack adds 9 detectors for real-time voice agents (dt.enable_pack("voice")). Built-in detectors always run; packs only add to them and start in shadow mode.

→ detector packs · voice pack · wiring a voice framework

For failure modes no structural check can catch — did the agent hallucinate, did it finish the task, did it solve the wrong task, is the user going in circles. Post-hoc (never in your agent's request path), sampling-based, disabled by default. Ships seven DeepEval-backed evaluators — four run-level (hallucination, task completion, task-understanding failure, off-topic drift) and three conversation-level (user frustration, confusion loops, sycophancy) — plus false-positive management (confidence floors, grouping, feedback loop, second-opinion for high-stakes findings). Each calibrated before ship (see scripts/calibration/).

SEMANTIC_WORKER_ENABLED=true

The semantic worker runs as its own container. It's in both compose files and off by default — set the flag in .env and bring the stack back up. Same for the external-evaluation and ElevenLabs workers.

→ docs/semantic-evaluation.md

Live at http://localhost:3000. Auto-refreshes every 15s.

→ docs/dashboard.md

Slack and generic webhook (PagerDuty, Linear, custom).

SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
SLACK_MIN_SEVERITY=LOW   # LOW | MEDIUM | HIGH | CRITICAL

A weekly digest (Monday 9am UTC) summarises top failure types and systemic patterns. Enable with DIGEST_ENABLED=true.

→ docs/alerts.md

Root-cause analysis is native — no third-party tracer required. Click Explain + on any alert and Dunetrace analyzes the run's own stored events and returns a specific cause and fix. Every fix is one of two kinds:

• Policy fixes (tool loops, retry storms, runaway step counts) → Dunetrace applies a runtime guardrail directly, no code change needed
• Prompt / code fixes → a diff you copy in, or a one-click draft PR on GitHub for code/infra changes

Fix effectiveness is tracked automatically.

Runtime guardrails that fire mid-run — before a failure propagates.

dt.add_policy(
    name="cap tool calls",
    condition={"trigger": "tool_call_count", "operator": "gt", "value": 5},
    action={"type": "stop"},
)
dt.add_policy(
    name="cost cap",
    condition={"trigger": "cost_usd", "operator": "gt", "value": 0.50},
    action={"type": "switch_model", "params": {"model": "gpt-4o-mini"}},
)

Policies can also be created in the dashboard and fetched automatically by the SDK (60s TTL).

Human-in-the-loop approvals — gate a risky tool (wiring money, deleting data) behind human approval. A require_approval policy blocks the tool call until someone approves in Slack or the dashboard, or it times out (fail-closed: a timeout blocks the tool). No agent code changes — the gate fires on the existing tool-call hook.

dt.add_policy(
    name="approve-wires",
    condition={"trigger": "before_tool_call", "operator": "eq", "value": "wire_money"},
    action={"type": "require_approval", "params": {"timeout_s": 300}},
)

→ docs/policies.md · docs/approvals.md

Query agent signals directly from Claude Code, Cursor, or Codex — without leaving your editor.

pip install dunetrace-mcp

Claude Code: registered automatically in ~/.claude.json after pip install dunetrace-mcp. Restart Claude Code to load.

Cursor: add .cursor/mcp.json to your project root:

{
  "mcpServers": {
    "dunetrace": {
      "command": "dunetrace-mcp",
      "env": {
        "DUNETRACE_API_URL": "http://localhost:8002",
        "DUNETRACE_API_KEY": "dt_dev_test"
      }
    }
  }
}

→ docs/mcp-server.md

Agent Code
  └─► Dunetrace SDK        (raw content → ingest events)
        └─► Ingest API      (POST /v1/ingest → Postgres)
                ├─► Detector          (poll → 29 detectors → signals)
                ├─► Semantic Worker   (optional — poll → DeepEval → signals)
                ├─► Integrations      (optional — pull Langfuse/LangSmith/Braintrust)
                ├─► Alerts            (poll → explain → Slack / webhook)
                └─► Customer API      (runs, signals, explanations → dashboard)

→ docs/architecture.md for the full service breakdown · operations guide (retention, rate limiting, quotas)

Evaluation & tracing

• OpenTelemetry export (Datadog, Grafana, Honeycomb, Signoz, any OTLP backend)
• OpenTelemetry ingestion (send your existing OTel traces to Dunetrace: OpenLIT, Traceloop, OTel contrib)
• External evaluation (Langfuse, LangSmith, Braintrust, generic push)
• Semantic evaluation (Dunetrace's own DeepEval-backed layer)

Fix & workflow

• GitHub App (automated draft PRs)
• Slack & Linear alerts
• Coding agents — Claude Code, Cursor, Codex (MCP)

Voice

• ElevenLabs (correlate TTS cost and voice choices with agent behavior)
• Wiring a voice framework
• Voice metrics (call-level view of a voice agent)
• Voice detector pack (9 detectors)

Fork, branch, change, make test, PR. For larger changes (new integrations, architecture changes), open an issue first.

New here? See CONTRIBUTING.md for setup and workflow, browse the good first issues, and if required, follow the step-by-step Adding a detector guide.

Requires Python 3.11+, Node.js 22+, Docker + Docker Compose.

dunetrace@gmail.com

Apache 2.0