Add support for per-prompt execution timeout (prompt_timeout_seconds)#493
Add support for per-prompt execution timeout (prompt_timeout_seconds)#493graceqi-g wants to merge 4 commits into
Conversation
2e3a7e9 to
0c78054
Compare
|
/gcbrun |
|
/gcbrun |
| timed_out = True | ||
| proc.kill() | ||
| break | ||
| line = proc.stdout.readline() |
There was a problem hiding this comment.
proc.stdout.readline() is a blocking call. If the subprocess stalls mid-line (never emits a newline) rather than between lines, the timeout check on the while True loop never gets revisited until the next line arrives the timeout is only enforced between lines, not truly wall-clock bounded. Consider using non-blocking reads (e.g., select, a reader thread with a queue, or proc.stdout.readline wrapped with signal.alarm/asyncio) to guarantee the kill happens within timeout regardless of the subprocess's I/O behavior.
There was a problem hiding this comment.
added threading.Timer to ensure the timeout is enforced
5e97d01 to
3d141aa
Compare
fb3eed0 to
bbfa5d2
Compare
|
/gcbrun |
Supports setting prompt_timeout_seconds across Agent CLI generators (gemini_cli, claude_code, codex_cli, agy_cli) to terminate prompt turns that spin indefinitely. Configurable per scenario JSON, per model YAML, or per run_config YAML. TAG=agy CONV=4229555b-deeb-44bb-a481-c751dab59476
bbfa5d2 to
aba996d
Compare
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
Neither subprocess.run(..., timeout=...) (gemini/claude/agy) nor proc.kill() (codex) kills the process group — only the direct child. These agent CLIs are node-based and routinely fork subprocess trees (MCP servers, tool subprocesses). On timeout, grandchildren can be orphaned and keep running, holding ports/CPU/memory. Across a long eval run with repeated timeouts, this can accumulate |
|
subprocess.TimeoutExpired carries the output captured so far on e.stdout / e.stderr, but the handlers hardcode "": This is inconsistent with the Codex path, which preserves stdout_lines. More importantly, it contradicts the existing design intent in agy_cli._run_agy_cli, whose own comment says "a timed-out/errored run still ends in a result event carrying real data" — but with a hard subprocess.run timeout, that stream is thrown away, so the partial trajectory/tool calls before the hang are lost. |
Supports setting prompt_timeout_seconds across Agent CLI generators (gemini_cli, claude_code, codex_cli, agy_cli) to terminate prompt turns that spin indefinitely. Configurable per scenario JSON, per model YAML, or per run_config YAML.
Order of precedence from highest priority to lowest is scenario, model, then run config.