Skip to content

Release orchestrator reads .ci.yaml/computes engine hash from HEAD, regardless of commit #174158

@matanlurey

Description

@matanlurey

Discovered while debugging #174155, which is why recipes/+/68803 has not been reverted yet.


@jtmcdole and I accidentally discovered that release_builder.py/GetRepoInfo is wrong - it always fetches (and implicitly, checks out) the HEAD of the given release branch, not the current commit. This causes inconsistent builds if there are multiple cherrypicks merged.

Imagine the following, where we've merged 3 cherrypicks into flutter-3.36-candidate.0:

⬜ abc104 -- HEAD of flutter-3.36-candidate.0
⬜ abc103
🟨 abc102
🟩 abc101

When running a (release) engine (and docs? I think?) build for abc102, we do the following:

  1. Initialize Git (OK)
Executing command [
  '/b/s/w/ir/cipd_bin_packages/cpython3/bin/python3',
  '-u',
  '/b/s/w/ir/kitchen-checkout/depot_tools/recipes/recipe_modules/git/resources/git_setup.py',
  '--path',
  '/b/s/w/ir/x/w/mirrors/flutter',
  '--url',
  'https://flutter.googlesource.com/mirrors/flutter',
]
  1. Fetch the HEAD of the branch:
Executing command [
  '/b/s/w/ir/cipd_bin_packages/git',
  'fetch',
  'origin',
  'flutter-3.35-candidate.0',
  '--recurse-submodules',
  '--progress',
  '--tags',
]
  1. Checkout the HEAD of the branch (Wrong!)
Executing command [
  '/b/s/w/ir/cipd_bin_packages/git',
  'checkout',
  '-f',
  'FETCH_HEAD',
]

That means, in the above example, abc102 checks out (and builds) abc104.

There seems to be a conflation between the term git_ref - it sometimes (elsewhere) means "the current SHA" (and indeed, even in the recipe it's documented as * The git ref to check out) - but in this recipe it's actually returning the branch name, and indeed that is what the gitiles data that is being passed to the builder contains:

{
  "$recipe_engine/scheduler": {
    "hostname": "luci-scheduler.appspot.com",
    "invocation": "8871440901813659728",
    "job": "dart-internal/Linux flutter_release_builder",
    "triggers": [
      {
        "gitiles": {
          "ref": "refs/heads/flutter-3.35-candidate.0",
          "repo": "https://flutter.googlesource.com/mirrors/flutter",
          "revision": "03e97fb327ab9c9ebdb0d0928e4735cdc7c7c0e2"
        },

Metadata

Metadata

Assignees

Labels

P0Critical issues such as a build break or regressionfyi-engineFor the attention of Engine teaminfra: recipesIssues with flutter/recipesinfra: releaseRelease-related requests and toolingteam-infraOwned by Infrastructure teamtriaged-infraTriaged by Infrastructure team

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions