-
-
Notifications
You must be signed in to change notification settings - Fork 791
fix(runner): prevent retry immediately race condition which can cause stuck runs that end up being system failures #2402
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
… stuck runs that end up being system failures
🦋 Changeset detectedLatest commit: a70328d The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Walkthrough
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. ✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
🧹 Nitpick comments (4)
.changeset/loud-rules-dream.md (1)
5-5
: Polish the changeset description grammar for clarityMinor grammar/wording tweak makes this clearer and avoids redundancy.
-fix(runner): prevent retry immediately race condition which can cause stuck runs that end up being system failures +fix(runner): prevent retry-immediately race condition that can cause stuck runs and system failurespackages/cli-v3/src/entryPoints/managed/execution.ts (3)
80-80
: Make snapshot-change suppression reentrancy-safe (counter vs boolean)Using a boolean here can be fragile if
enableIgnoreSnapshotChanges
is ever called nested (now or in the future) — the outer call’sfinally
would flip it back to false too early. A lightweight counter avoids that footgun.- private ignoreSnapshotChanges = false; + private ignoreSnapshotChangesDepth = 0;Follow-up diffs for the guard and helper are provided in comments below.
241-249
: Guard looks good; make it work with a counter and enrich logs with runIdTwo tweaks:
- If you adopt the counter, check
> 0
here.- Add
runId
for faster correlation in logs.- if (this.ignoreSnapshotChanges) { - this.sendDebugLog("processSnapshotChange: ignoring snapshot change", { - incomingSnapshotId: snapshot.friendlyId, - completedWaitpoints: completedWaitpoints.length, - currentAttemptNumber: this.currentAttemptNumber, - newAttemptNumber: run.attemptNumber, - }); + if (this.ignoreSnapshotChangesDepth > 0) { + this.sendDebugLog("processSnapshotChange: ignoring snapshot change", { + incomingSnapshotId: snapshot.friendlyId, + completedWaitpoints: completedWaitpoints.length, + currentAttemptNumber: this.currentAttemptNumber, + newAttemptNumber: run.attemptNumber, + runId: run.friendlyId, + }); return; }
845-852
: Refactor helper to support nesting (pairs with the counter change above)This change makes suppression robust if nested calls ever occur.
- private async enableIgnoreSnapshotChanges<T>(fn: () => Promise<T>): Promise<T> { - this.ignoreSnapshotChanges = true; - try { - return await fn(); - } finally { - this.ignoreSnapshotChanges = false; - } - } + private async enableIgnoreSnapshotChanges<T>(fn: () => Promise<T>): Promise<T> { + this.ignoreSnapshotChangesDepth++; + try { + return await fn(); + } finally { + this.ignoreSnapshotChangesDepth--; + } + }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
⛔ Files ignored due to path filters (1)
references/hello-world/src/trigger/attemptFailures.ts
is excluded by!references/**
📒 Files selected for processing (2)
.changeset/loud-rules-dream.md
(1 hunks)packages/cli-v3/src/entryPoints/managed/execution.ts
(4 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.{ts,tsx}
📄 CodeRabbit Inference Engine (.github/copilot-instructions.md)
**/*.{ts,tsx}
: Always prefer using isomorphic code like fetch, ReadableStream, etc. instead of Node.js specific code
For TypeScript, we usually use types over interfaces
Avoid enums
No default exports, use function declarations
Files:
packages/cli-v3/src/entryPoints/managed/execution.ts
🧬 Code Graph Analysis (1)
packages/cli-v3/src/entryPoints/managed/execution.ts (1)
packages/core/src/utils.ts (1)
tryCatch
(5-18)
🪛 LanguageTool
.changeset/loud-rules-dream.md
[grammar] ~5-~5: There might be a mistake here.
Context: ...k runs that end up being system failures
(QB_NEW_EN)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (23)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (7, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (2, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (6, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (6, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (5, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (8, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (4, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (1, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (3, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (8, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (5, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (4, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (7, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (2, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (1, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (3, 8)
- GitHub Check: units / packages / 🧪 Unit Tests: Packages (1, 1)
- GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
- GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
- GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
- GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
- GitHub Check: typecheck / typecheck
- GitHub Check: Analyze (javascript-typescript)
No description provided.