Skip to content

Fix race condition in workspace build API endpoint #18948

@blink-so

Description

@blink-so

Problem

As identified in PR #18932, there's a race condition in the workspace build API endpoint that can cause flaky tests and potentially inconsistent API responses.

The issue occurs in the /api/v2/workspacesbuilds/{workspacebuild}/ endpoint where two separate database queries happen outside of a transaction:

  1. Query for database.WorkspaceBuild in the route handler
  2. Call to GetProvisionerJobsByIDsWithQueuePosition in workspaceBuildsData

Because these calls happen outside of a transaction, the state can change between them, resulting in an in-progress workspace build having a completed provisioner job attached to it.

Code References

  • Route handler:

    coder/coderd/coderd.go

    Lines 1409 to 1415 in a3f64f7

    r.Route("/workspacebuilds/{workspacebuild}", func(r chi.Router) {
    r.Use(
    apiKeyMiddleware,
    httpmw.ExtractWorkspaceBuildParam(options.Database),
    httpmw.ExtractWorkspaceParam(options.Database),
    )
    r.Get("/", api.workspaceBuild)
  • workspaceBuildsData call:
    data, err := api.workspaceBuildsData(ctx, []database.WorkspaceBuild{workspaceBuild})
  • GetProvisionerJobsByIDsWithQueuePosition call:
    jobs, err := api.Database.GetProvisionerJobsByIDsWithQueuePosition(ctx, database.GetProvisionerJobsByIDsWithQueuePositionParams{
    IDs: jobIDs,
    StaleIntervalMS: provisionerdserver.StaleInterval.Milliseconds(),
    })
    if err != nil && !errors.Is(err, sql.ErrNoRows) {

Solution

The database queries should be wrapped in a transaction to ensure consistency, or the logic should be restructured to avoid the race condition.

Impact

  • Fixes flaky test TestAPI/ModifyAutostopWithRunningWorkspace
  • May fix other similar flakes in the test suite
  • Improves API consistency and reliability

Follow-up work identified from PR #18932

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions