Skip to content

Reduce false error logging (round 1) #2415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Aug 19, 2025
Merged

Conversation

matt-aitken
Copy link
Member

No description provided.

Copy link

changeset-bot bot commented Aug 18, 2025

⚠️ No Changeset found

Latest commit: f9ddc86

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link
Contributor

coderabbitai bot commented Aug 18, 2025

Walkthrough

Changes adjust logging severities across alert delivery, task finalization, and run-engine systems. In Slack alert delivery, several errors are downgraded (error→warn), and a new check treats Slack “account_inactive” as a SkipRetryError with info-level log and early exit. FinalizeTaskRun downgrades fatal-status logs (error→warn) and logs missing lockedById at info with added status context. CheckpointSystem logs invalid snapshot at info. DequeueSystem replaces a single error with conditional error/warn based on a new isExecuting helper, which is now exported from statuses.js. Control flow remains the same aside from the new account_inactive skip-retry path.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch reduce-false-error-logging

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🔭 Outside diff range comments (1)
internal-packages/run-engine/src/engine/systems/checkpointSystem.ts (1)

117-133: Transaction leakage: using this.$.prisma inside a tx-scoped method bypasses the provided transaction

Both createCheckpoint() and continueRunExecution() derive a local prisma = tx ?? this.$.prisma but then call this.$.prisma.taskRun.update(...). This escapes the transaction when tx is provided, risking inconsistent state and race conditions.

Apply these diffs to use the tx-scoped client:

-      const run = await this.$.prisma.taskRun.update({
+      const run = await prisma.taskRun.update({
         where: {
           id: runId,
         },
         data: {
           status: "WAITING_TO_RESUME",
         },
         include: {
           runtimeEnvironment: {
             include: {
               project: true,
               organization: true,
             },
           },
         },
       });
-      const run = await this.$.prisma.taskRun.update({
+      const run = await prisma.taskRun.update({
         where: {
           id: runId,
         },
         data: {
           status: "EXECUTING",
         },
         select: {
           id: true,
           status: true,
           attemptNumber: true,
           organizationId: true,
           runtimeEnvironmentId: true,
           projectId: true,
           updatedAt: true,
           createdAt: true,
         },
       });

Also applies to: 295-312

🧹 Nitpick comments (6)
internal-packages/run-engine/src/engine/systems/checkpointSystem.ts (2)

65-68: Trim logged payload to avoid large/PII-heavy logs

Logging the whole snapshot can be verbose and may include sensitive/large metadata. Consider logging only identifiers and status.

Apply this diff:

-        this.$.logger.info("Tried to createCheckpoint on an invalid snapshot", {
-          snapshot,
-          snapshotId,
-        });
+        this.$.logger.info("Tried to createCheckpoint on an invalid snapshot", {
+          snapshotId,
+          latestSnapshotId: snapshot.id,
+          executionStatus: snapshot.executionStatus,
+        });

315-318: Incorrect log message references createCheckpoint in continueRunExecution

The message says "Run not found for createCheckpoint" but this is inside continueRunExecution(), which can mislead ops triage.

Apply this diff:

-        this.$.logger.error("Run not found for createCheckpoint", {
+        this.$.logger.error("Run not found for continueRunExecution", {
           snapshot,
         });
internal-packages/run-engine/src/engine/systems/dequeueSystem.ts (2)

136-144: Prefer structured logging over string interpolation

Switching to structured objects improves queryability in log aggregation (filter by runId/snapshotId/status) and avoids embedded newlines.

Apply this diff:

-                if (isExecuting(snapshot.executionStatus)) {
-                  this.$.logger.error(
-                    `RunEngine.dequeueFromWorkerQueue(): Run is not in a valid state to be dequeued: ${runId}\n ${snapshot.id}:${snapshot.executionStatus}`
-                  );
-                } else {
-                  this.$.logger.warn(
-                    `RunEngine.dequeueFromWorkerQueue(): Run is in an expected not valid state to be dequeued: ${runId}\n ${snapshot.id}:${snapshot.executionStatus}`
-                  );
-                }
+                if (isExecuting(snapshot.executionStatus)) {
+                  this.$.logger.error(
+                    "RunEngine.dequeueFromWorkerQueue(): Run is not in a valid state to be dequeued",
+                    {
+                      runId,
+                      snapshotId: snapshot.id,
+                      executionStatus: snapshot.executionStatus,
+                    }
+                  );
+                } else {
+                  this.$.logger.warn(
+                    "RunEngine.dequeueFromWorkerQueue(): Run is in an expected not-valid state to be dequeued",
+                    {
+                      runId,
+                      snapshotId: snapshot.id,
+                      executionStatus: snapshot.executionStatus,
+                    }
+                  );
+                }

88-96: Nit: log message still says dequeueFromMasterQueue

This method is dequeueFromWorkerQueue; keeping log identifiers consistent helps debugging.

Apply this diff:

-                  this.$.logger.error(
-                    "RunEngine.dequeueFromMasterQueue(): Run is already PENDING_EXECUTING, removing from queue",
+                  this.$.logger.error(
+                    "RunEngine.dequeueFromWorkerQueue(): Run is already PENDING_EXECUTING, removing from queue",
                     {
                       runId,
                       orgId,
                     }
                   );
apps/webapp/app/v3/services/alerts/deliverAlert.server.ts (2)

955-959: Include Slack retryAfter in logs for rate limiting

Adding retryAfter makes SRE decisions easier and could inform backoff strategies.

Apply this diff:

-        logger.warn("[DeliverAlert] Slack rate limited", {
-          error,
-          message,
-        });
+        logger.warn("[DeliverAlert] Slack rate limited", {
+          error,
+          message,
+          retryAfter: error.retryAfter,
+        });

Optionally, consider re-enqueuing the alert with a delay of retryAfter seconds to avoid immediate retries.

I can draft a small helper to nack/re-enqueue with delay using alertsWorker if you'd like.


981-985: Platform error downgraded to warn — consider classifying more non-retriable codes

You already special-case invalid_blocks and account_inactive. Other common non-retriables (e.g., channel_not_found, is_archived, not_in_channel, token_revoked, invalid_auth, not_authed) could also skip retries to further reduce noise.

Apply this diff to extend skip logic:

       if (isWebAPIPlatformError(error)) {
         logger.warn("[DeliverAlert] Slack platform error", {
           error,
           message,
         });

         if (error.data.error === "invalid_blocks") {
           logger.error("[DeliverAlert] Slack invalid blocks", {
             error,
           });

           throw new SkipRetryError("Slack invalid blocks");
         }

-        if (error.data.error === "account_inactive") {
+        const NON_RETRIABLE_PLATFORM_ERROR_CODES = new Set([
+          "account_inactive",
+          "channel_not_found",
+          "is_archived",
+          "not_in_channel",
+          "token_revoked",
+          "invalid_auth",
+          "not_authed",
+        ]);
+
+        if (NON_RETRIABLE_PLATFORM_ERROR_CODES.has(error.data.error)) {
           logger.info("[DeliverAlert] Slack account inactive, skipping retry", {
             error,
             message,
           });

-          throw new SkipRetryError("Slack account inactive");
+          throw new SkipRetryError(`Slack non-retriable platform error: ${error.data.error}`);
         }

If you'd prefer, I can scope this to a smaller subset.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 471c960 and f9ddc86.

📒 Files selected for processing (4)
  • apps/webapp/app/v3/services/alerts/deliverAlert.server.ts (6 hunks)
  • apps/webapp/app/v3/services/finalizeTaskRun.server.ts (3 hunks)
  • internal-packages/run-engine/src/engine/systems/checkpointSystem.ts (1 hunks)
  • internal-packages/run-engine/src/engine/systems/dequeueSystem.ts (2 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
**/*.{ts,tsx}

📄 CodeRabbit Inference Engine (.github/copilot-instructions.md)

**/*.{ts,tsx}: Always prefer using isomorphic code like fetch, ReadableStream, etc. instead of Node.js specific code
For TypeScript, we usually use types over interfaces
Avoid enums
No default exports, use function declarations

Files:

  • internal-packages/run-engine/src/engine/systems/checkpointSystem.ts
  • apps/webapp/app/v3/services/alerts/deliverAlert.server.ts
  • internal-packages/run-engine/src/engine/systems/dequeueSystem.ts
  • apps/webapp/app/v3/services/finalizeTaskRun.server.ts
{packages/core,apps/webapp}/**/*.{ts,tsx}

📄 CodeRabbit Inference Engine (.github/copilot-instructions.md)

We use zod a lot in packages/core and in the webapp

Files:

  • apps/webapp/app/v3/services/alerts/deliverAlert.server.ts
  • apps/webapp/app/v3/services/finalizeTaskRun.server.ts
apps/webapp/**/*.{ts,tsx}

📄 CodeRabbit Inference Engine (.cursor/rules/webapp.mdc)

apps/webapp/**/*.{ts,tsx}: In the webapp, all environment variables must be accessed through the env export of env.server.ts, instead of directly accessing process.env.
When importing from @trigger.dev/core in the webapp, never import from the root @trigger.dev/core path; always use one of the subpath exports as defined in the package's package.json.

Files:

  • apps/webapp/app/v3/services/alerts/deliverAlert.server.ts
  • apps/webapp/app/v3/services/finalizeTaskRun.server.ts
🧬 Code Graph Analysis (1)
internal-packages/run-engine/src/engine/systems/dequeueSystem.ts (1)
internal-packages/run-engine/src/engine/statuses.ts (1)
  • isExecuting (8-14)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (23)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (7, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (8, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (5, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (6, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (8, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (3, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (2, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (4, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (7, 8)
  • GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (1, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (5, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (6, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (3, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (4, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (1, 8)
  • GitHub Check: units / internal / 🧪 Unit Tests: Internal (2, 8)
  • GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
  • GitHub Check: units / packages / 🧪 Unit Tests: Packages (1, 1)
  • GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
  • GitHub Check: typecheck / typecheck
  • GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (10)
internal-packages/run-engine/src/engine/systems/checkpointSystem.ts (1)

65-68: Info-level log for invalid snapshot is appropriate here

This reduces noisy error logs for an expected boundary condition and still preserves telemetry via the discard event. Good change.

internal-packages/run-engine/src/engine/systems/dequeueSystem.ts (1)

11-11: Importing isExecuting is a solid addition

This enables differentiated logging for truly executing vs other invalid states, aligning with the PR goal.

apps/webapp/app/v3/services/finalizeTaskRun.server.ts (3)

153-154: Downgrading fatal-status log to warn is appropriate

Keeps noise down while preserving signal. The additional status in the payload is helpful.


172-176: Warn for fatal status in non-development environments looks good

Maintains visibility without over-indexing on errors.


308-312: Info-level for missing lockedById with clarifying comment is a good change

This is an expected scenario (expired/cancelled before attempt). The added status context helps triage.

apps/webapp/app/v3/services/alerts/deliverAlert.server.ts (5)

161-164: Warn when skipping retry is on point

This reduces false error noise while still surfacing the reason.


963-969: Warn on Slack HTTP error is consistent with the objective

Behavior remains the same while logging is less noisy.


972-978: Warn on Slack request error is appropriate

Keeps error logs cleaner for transient network issues.


994-1001: Info + SkipRetry on Slack account_inactive is the right call

This will reduce churn on alerts that can never succeed due to the account state.


1006-1012: Warn for generic Slack send failure is acceptable

You surface the original error and rethrow, preserving retry behavior.

@matt-aitken matt-aitken merged commit 0ee4234 into main Aug 19, 2025
31 checks passed
@matt-aitken matt-aitken deleted the reduce-false-error-logging branch August 19, 2025 08:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants