Skip to content

fix(runner): prevent retry immediately race condition which can cause stuck runs that end up being system failures #2402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 15, 2025

Conversation

ericallam
Copy link
Member

No description provided.

… stuck runs that end up being system failures
Copy link

changeset-bot bot commented Aug 15, 2025

🦋 Changeset detected

Latest commit: a70328d

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copy link
Contributor

coderabbitai bot commented Aug 15, 2025

Walkthrough

  • Adds a changeset entry documenting a patch release for Trigger.dev with a runner fix.
  • In RunExecution (packages/cli-v3/src/entryPoints/managed/execution.ts), introduces a private boolean flag ignoreSnapshotChanges.
  • Adds a guard in processSnapshotChange to early-return when ignoreSnapshotChanges is true and logs debug details.
  • Adds helper enableIgnoreSnapshotChanges to set/reset the flag around an async function.
  • Wraps startAttempt calls in execute() (warm start path) and retryImmediately() (post-delay) with enableIgnoreSnapshotChanges.
  • Enhances debug logging to include snapshot and attempt details.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fix/retry-immediately-system-failures

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (4)
.changeset/loud-rules-dream.md (1)

5-5: Polish the changeset description grammar for clarity

Minor grammar/wording tweak makes this clearer and avoids redundancy.

-fix(runner): prevent retry immediately race condition which can cause stuck runs that end up being system failures
+fix(runner): prevent retry-immediately race condition that can cause stuck runs and system failures
packages/cli-v3/src/entryPoints/managed/execution.ts (3)

80-80: Make snapshot-change suppression reentrancy-safe (counter vs boolean)

Using a boolean here can be fragile if enableIgnoreSnapshotChanges is ever called nested (now or in the future) — the outer call’s finally would flip it back to false too early. A lightweight counter avoids that footgun.

-  private ignoreSnapshotChanges = false;
+  private ignoreSnapshotChangesDepth = 0;

Follow-up diffs for the guard and helper are provided in comments below.


241-249: Guard looks good; make it work with a counter and enrich logs with runId

Two tweaks:

  • If you adopt the counter, check > 0 here.
  • Add runId for faster correlation in logs.
-    if (this.ignoreSnapshotChanges) {
-      this.sendDebugLog("processSnapshotChange: ignoring snapshot change", {
-        incomingSnapshotId: snapshot.friendlyId,
-        completedWaitpoints: completedWaitpoints.length,
-        currentAttemptNumber: this.currentAttemptNumber,
-        newAttemptNumber: run.attemptNumber,
-      });
+    if (this.ignoreSnapshotChangesDepth > 0) {
+      this.sendDebugLog("processSnapshotChange: ignoring snapshot change", {
+        incomingSnapshotId: snapshot.friendlyId,
+        completedWaitpoints: completedWaitpoints.length,
+        currentAttemptNumber: this.currentAttemptNumber,
+        newAttemptNumber: run.attemptNumber,
+        runId: run.friendlyId,
+      });
       return;
     }

845-852: Refactor helper to support nesting (pairs with the counter change above)

This change makes suppression robust if nested calls ever occur.

-  private async enableIgnoreSnapshotChanges<T>(fn: () => Promise<T>): Promise<T> {
-    this.ignoreSnapshotChanges = true;
-    try {
-      return await fn();
-    } finally {
-      this.ignoreSnapshotChanges = false;
-    }
-  }
+  private async enableIgnoreSnapshotChanges<T>(fn: () => Promise<T>): Promise<T> {
+    this.ignoreSnapshotChangesDepth++;
+    try {
+      return await fn();
+    } finally {
+      this.ignoreSnapshotChangesDepth--;
+    }
+  }
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 816c7ad and a70328d.

⛔ Files ignored due to path filters (1)
  • references/hello-world/src/trigger/attemptFailures.ts is excluded by !references/**
📒 Files selected for processing (2)
  • .changeset/loud-rules-dream.md (1 hunks)
  • packages/cli-v3/src/entryPoints/managed/execution.ts (4 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.{ts,tsx}

📄 CodeRabbit Inference Engine (.github/copilot-instructions.md)

**/*.{ts,tsx}: Always prefer using isomorphic code like fetch, ReadableStream, etc. instead of Node.js specific code
For TypeScript, we usually use types over interfaces
Avoid enums
No default exports, use function declarations

Files:

  • packages/cli-v3/src/entryPoints/managed/execution.ts
🧬 Code Graph Analysis (1)
packages/cli-v3/src/entryPoints/managed/execution.ts (1)
packages/core/src/utils.ts (1)
  • tryCatch (5-18)
🪛 LanguageTool
.changeset/loud-rules-dream.md

[grammar] ~5-~5: There might be a mistake here.
Context: ...k runs that end up being system failures

(QB_NEW_EN)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (23)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (7, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (2, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (6, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (6, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (5, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (8, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (4, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (1, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (3, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (8, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (5, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (4, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (7, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (2, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (1, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (3, 8)
  • GitHub Check: units / packages / 🧪 Unit Tests: Packages (1, 1)
  • GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
  • GitHub Check: typecheck / typecheck
  • GitHub Check: Analyze (javascript-typescript)

@ericallam ericallam merged commit 1a88809 into main Aug 15, 2025
31 checks passed
@ericallam ericallam deleted the fix/retry-immediately-system-failures branch August 15, 2025 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants