Upload GN CIPD package This change always builds the CIPD package for GN for CI builds and when requested also uploads that package to CIPD server. Change-Id: Idc62ed9c6b4c491426b2c7e26b87cad03a206d5e Reviewed-on: https://gn-review.googlesource.com/1781 Reviewed-by: Andrii Shyshkalov <tandrii@google.com> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Reviewed-by: Scott Graham <scottmg@chromium.org> Commit-Queue: Petr Hosek <phosek@google.com>
diff --git a/infra/README.recipes.md b/infra/README.recipes.md index 9d447de..afa61e5 100644 --- a/infra/README.recipes.md +++ b/infra/README.recipes.md
@@ -35,24 +35,26 @@ ### *recipes* / [gn](/infra/recipes/gn.py) -[DEPS](/infra/recipes/gn.py#6): [windows\_sdk](#recipe_modules-windows_sdk), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/step][recipe_engine/recipe_modules/step] +[DEPS](/infra/recipes/gn.py#8): [windows\_sdk](#recipe_modules-windows_sdk), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step] Recipe for building GN. -— **def [RunSteps](/infra/recipes/gn.py#20)(api):** +— **def [RunSteps](/infra/recipes/gn.py#29)(api, repository):** ### *recipes* / [windows\_sdk:examples/full](/infra/recipe_modules/windows_sdk/examples/full.py) [DEPS](/infra/recipe_modules/windows_sdk/examples/full.py#5): [windows\_sdk](#recipe_modules-windows_sdk), [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step] — **def [RunSteps](/infra/recipe_modules/windows_sdk/examples/full.py#13)(api):** -[recipe_engine/recipe_modules/buildbucket]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-buildbucket -[recipe_engine/recipe_modules/cipd]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-cipd -[recipe_engine/recipe_modules/context]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-context -[recipe_engine/recipe_modules/json]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-json -[recipe_engine/recipe_modules/path]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-path -[recipe_engine/recipe_modules/platform]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-platform -[recipe_engine/recipe_modules/properties]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-properties -[recipe_engine/recipe_modules/python]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-python -[recipe_engine/recipe_modules/step]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-step -[recipe_engine/wkt/RecipeApi]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/recipe_engine/recipe_api.py#1006 +[recipe_engine/recipe_modules/buildbucket]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-buildbucket +[recipe_engine/recipe_modules/cipd]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-cipd +[recipe_engine/recipe_modules/context]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-context +[recipe_engine/recipe_modules/file]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-file +[recipe_engine/recipe_modules/json]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-json +[recipe_engine/recipe_modules/path]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-path +[recipe_engine/recipe_modules/platform]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-platform +[recipe_engine/recipe_modules/properties]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-properties +[recipe_engine/recipe_modules/python]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-python +[recipe_engine/recipe_modules/raw_io]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-raw_io +[recipe_engine/recipe_modules/step]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-step +[recipe_engine/wkt/RecipeApi]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/recipe_engine/recipe_api.py#1006
diff --git a/infra/config/recipes.cfg b/infra/config/recipes.cfg index d0130f8..9d8e189 100644 --- a/infra/config/recipes.cfg +++ b/infra/config/recipes.cfg
@@ -3,7 +3,7 @@ "deps": { "recipe_engine": { "branch": "master", - "revision": "40699af88c76d65ef1b3d364de221611603e2dc6", + "revision": "94bb0a758456a5df6891677a8cf2876c07ea5f0c", "url": "https://chromium.googlesource.com/infra/luci/recipes-py" } },
diff --git a/infra/recipes.py b/infra/recipes.py index fe6589d..d726c68 100755 --- a/infra/recipes.py +++ b/infra/recipes.py
@@ -107,7 +107,9 @@ raise MalformedRecipesCfg(ex.message, recipes_cfg_path) -GIT = 'git.bat' if sys.platform.startswith(('win', 'cygwin')) else 'git' +_BAT = '.bat' if sys.platform.startswith(('win', 'cygwin')) else '' +GIT = 'git' + _BAT +VPYTHON = 'vpython' + _BAT def _subprocess_call(argv, **kwargs): @@ -208,7 +210,7 @@ engine_path = checkout_engine(engine_override, repo_root, recipes_cfg_path) return _subprocess_call([ - sys.executable, '-u', + VPYTHON, '-u', os.path.join(engine_path, 'recipes.py')] + args)
diff --git a/infra/recipes/gn.expected/ci_win.json b/infra/recipes/gn.expected/ci.json similarity index 91% rename from infra/recipes/gn.expected/ci_win.json rename to infra/recipes/gn.expected/ci.json index 75e7300..d6ce20e 100644 --- a/infra/recipes/gn.expected/ci_win.json +++ b/infra/recipes/gn.expected/ci.json
@@ -345,6 +345,28 @@ ] }, { + "cmd": [ + "cipd.bat", + "pkg-build", + "-pkg-def", + "{\"data\": [{\"file\": \"gn.exe\"}, {\"version_file\": \".versions/gn.exe.cipd_version\"}], \"install_mode\": \"copy\", \"package\": \"gn/gn/${platform}\", \"root\": \"[START_DIR]\\\\gn\\\\out\"}", + "-out", + "[CLEANUP]\\gn.cipd", + "-json-output", + "/path/to/tmp/json" + ], + "name": "build gn/gn/${platform}", + "~followup_annotations": [ + "@@@STEP_LOG_LINE@json.output@{@@@", + "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@", + "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"40-chars-fake-of-the-package-instance_id\", @@@", + "@@@STEP_LOG_LINE@json.output@ \"package\": \"gn/gn/${platform}\"@@@", + "@@@STEP_LOG_LINE@json.output@ }@@@", + "@@@STEP_LOG_LINE@json.output@}@@@", + "@@@STEP_LOG_END@json.output@@@" + ] + }, + { "name": "$result", "recipe_result": null, "status_code": 0
diff --git a/infra/recipes/gn.expected/ci_linux.json b/infra/recipes/gn.expected/ci_linux.json deleted file mode 100644 index b313dc9..0000000 --- a/infra/recipes/gn.expected/ci_linux.json +++ /dev/null
@@ -1,194 +0,0 @@ -[ - { - "cmd": [], - "name": "git" - }, - { - "cmd": [ - "git", - "init", - "[START_DIR]/gn" - ], - "infra_step": true, - "name": "git.init", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "git", - "fetch", - "https://gn.googlesource.com/gn", - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - ], - "cwd": "[START_DIR]/gn", - "infra_step": true, - "name": "git.fetch", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "git", - "checkout", - "FETCH_HEAD" - ], - "cwd": "[START_DIR]/gn", - "infra_step": true, - "name": "git.checkout", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "cipd", - "ensure", - "-root", - "[START_DIR]/cipd", - "-ensure-file", - "fuchsia/clang/${platform} goma\ninfra/ninja/${platform} version:1.8.2", - "-json-output", - "/path/to/tmp/json" - ], - "infra_step": true, - "name": "ensure_installed", - "~followup_annotations": [ - "@@@STEP_LOG_LINE@json.output@{@@@", - "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@", - "@@@STEP_LOG_LINE@json.output@ \"\": [@@@", - "@@@STEP_LOG_LINE@json.output@ {@@@", - "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"resolved-instance_id-of-goma------------\", @@@", - "@@@STEP_LOG_LINE@json.output@ \"package\": \"fuchsia/clang/${platform}\"@@@", - "@@@STEP_LOG_LINE@json.output@ }, @@@", - "@@@STEP_LOG_LINE@json.output@ {@@@", - "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"resolved-instance_id-of-version:1.8.2---\", @@@", - "@@@STEP_LOG_LINE@json.output@ \"package\": \"infra/ninja/${platform}\"@@@", - "@@@STEP_LOG_LINE@json.output@ }@@@", - "@@@STEP_LOG_LINE@json.output@ ]@@@", - "@@@STEP_LOG_LINE@json.output@ }@@@", - "@@@STEP_LOG_LINE@json.output@}@@@", - "@@@STEP_LOG_END@json.output@@@" - ] - }, - { - "cmd": [], - "name": "debug" - }, - { - "cmd": [], - "name": "debug.build", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "python", - "-u", - "[START_DIR]/gn/build/gen.py", - "-d" - ], - "cwd": "[START_DIR]/gn", - "env": { - "AR": "[START_DIR]/cipd/bin/llvm-ar", - "CC": "[START_DIR]/cipd/bin/clang", - "CXX": "[START_DIR]/cipd/bin/clang++", - "LDFLAGS": "-static-libstdc++ -ldl -lpthread" - }, - "name": "debug.build.generate", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@2@@@" - ] - }, - { - "cmd": [ - "[START_DIR]/cipd/ninja", - "-C", - "[START_DIR]/gn/out" - ], - "cwd": "[START_DIR]/gn", - "env": { - "AR": "[START_DIR]/cipd/bin/llvm-ar", - "CC": "[START_DIR]/cipd/bin/clang", - "CXX": "[START_DIR]/cipd/bin/clang++", - "LDFLAGS": "-static-libstdc++ -ldl -lpthread" - }, - "name": "debug.build.ninja", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@2@@@" - ] - }, - { - "cmd": [ - "[START_DIR]/gn/out/gn_unittests" - ], - "name": "debug.test", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [], - "name": "release" - }, - { - "cmd": [], - "name": "release.build", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "python", - "-u", - "[START_DIR]/gn/build/gen.py" - ], - "cwd": "[START_DIR]/gn", - "env": { - "AR": "[START_DIR]/cipd/bin/llvm-ar", - "CC": "[START_DIR]/cipd/bin/clang", - "CXX": "[START_DIR]/cipd/bin/clang++", - "LDFLAGS": "-static-libstdc++ -ldl -lpthread" - }, - "name": "release.build.generate", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@2@@@" - ] - }, - { - "cmd": [ - "[START_DIR]/cipd/ninja", - "-C", - "[START_DIR]/gn/out" - ], - "cwd": "[START_DIR]/gn", - "env": { - "AR": "[START_DIR]/cipd/bin/llvm-ar", - "CC": "[START_DIR]/cipd/bin/clang", - "CXX": "[START_DIR]/cipd/bin/clang++", - "LDFLAGS": "-static-libstdc++ -ldl -lpthread" - }, - "name": "release.build.ninja", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@2@@@" - ] - }, - { - "cmd": [ - "[START_DIR]/gn/out/gn_unittests" - ], - "name": "release.test", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "name": "$result", - "recipe_result": null, - "status_code": 0 - } -] \ No newline at end of file
diff --git a/infra/recipes/gn.expected/ci_mac.json b/infra/recipes/gn.expected/ci_mac.json deleted file mode 100644 index c4f9847..0000000 --- a/infra/recipes/gn.expected/ci_mac.json +++ /dev/null
@@ -1,166 +0,0 @@ -[ - { - "cmd": [], - "name": "git" - }, - { - "cmd": [ - "git", - "init", - "[START_DIR]/gn" - ], - "infra_step": true, - "name": "git.init", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "git", - "fetch", - "https://gn.googlesource.com/gn", - "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - ], - "cwd": "[START_DIR]/gn", - "infra_step": true, - "name": "git.fetch", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "git", - "checkout", - "FETCH_HEAD" - ], - "cwd": "[START_DIR]/gn", - "infra_step": true, - "name": "git.checkout", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "cipd", - "ensure", - "-root", - "[START_DIR]/cipd", - "-ensure-file", - "infra/ninja/${platform} version:1.8.2", - "-json-output", - "/path/to/tmp/json" - ], - "infra_step": true, - "name": "ensure_installed", - "~followup_annotations": [ - "@@@STEP_LOG_LINE@json.output@{@@@", - "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@", - "@@@STEP_LOG_LINE@json.output@ \"\": [@@@", - "@@@STEP_LOG_LINE@json.output@ {@@@", - "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"resolved-instance_id-of-version:1.8.2---\", @@@", - "@@@STEP_LOG_LINE@json.output@ \"package\": \"infra/ninja/${platform}\"@@@", - "@@@STEP_LOG_LINE@json.output@ }@@@", - "@@@STEP_LOG_LINE@json.output@ ]@@@", - "@@@STEP_LOG_LINE@json.output@ }@@@", - "@@@STEP_LOG_LINE@json.output@}@@@", - "@@@STEP_LOG_END@json.output@@@" - ] - }, - { - "cmd": [], - "name": "debug" - }, - { - "cmd": [], - "name": "debug.build", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "python", - "-u", - "[START_DIR]/gn/build/gen.py", - "-d" - ], - "cwd": "[START_DIR]/gn", - "name": "debug.build.generate", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@2@@@" - ] - }, - { - "cmd": [ - "[START_DIR]/cipd/ninja", - "-C", - "[START_DIR]/gn/out" - ], - "cwd": "[START_DIR]/gn", - "name": "debug.build.ninja", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@2@@@" - ] - }, - { - "cmd": [ - "[START_DIR]/gn/out/gn_unittests" - ], - "name": "debug.test", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [], - "name": "release" - }, - { - "cmd": [], - "name": "release.build", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "python", - "-u", - "[START_DIR]/gn/build/gen.py" - ], - "cwd": "[START_DIR]/gn", - "name": "release.build.generate", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@2@@@" - ] - }, - { - "cmd": [ - "[START_DIR]/cipd/ninja", - "-C", - "[START_DIR]/gn/out" - ], - "cwd": "[START_DIR]/gn", - "name": "release.build.ninja", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@2@@@" - ] - }, - { - "cmd": [ - "[START_DIR]/gn/out/gn_unittests" - ], - "name": "release.test", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "name": "$result", - "recipe_result": null, - "status_code": 0 - } -] \ No newline at end of file
diff --git a/infra/recipes/gn.expected/cq_linux.json b/infra/recipes/gn.expected/cipd_exists.json similarity index 71% rename from infra/recipes/gn.expected/cq_linux.json rename to infra/recipes/gn.expected/cipd_exists.json index b974d7b..c63252d 100644 --- a/infra/recipes/gn.expected/cq_linux.json +++ b/infra/recipes/gn.expected/cipd_exists.json
@@ -20,7 +20,7 @@ "git", "fetch", "https://gn.googlesource.com/gn", - "refs/heads/master" + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" ], "cwd": "[START_DIR]/gn", "infra_step": true, @@ -44,33 +44,6 @@ }, { "cmd": [ - "git", - "fetch", - "https://gn-review.googlesource.com/gn", - "refs/changes/00/1000/1" - ], - "cwd": "[START_DIR]/gn", - "infra_step": true, - "name": "git.fetch 1000/1", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "git", - "cherry-pick", - "FETCH_HEAD" - ], - "cwd": "[START_DIR]/gn", - "infra_step": true, - "name": "git.cherry-pick 1000/1", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ "cipd", "ensure", "-root", @@ -214,6 +187,65 @@ ] }, { + "cmd": [ + "cipd", + "pkg-build", + "-pkg-def", + "{\"data\": [{\"file\": \"gn\"}, {\"version_file\": \".versions/gn.cipd_version\"}], \"install_mode\": \"copy\", \"package\": \"gn/gn/${platform}\", \"root\": \"[START_DIR]/gn/out\"}", + "-out", + "[CLEANUP]/gn.cipd", + "-json-output", + "/path/to/tmp/json" + ], + "name": "build gn/gn/${platform}", + "~followup_annotations": [ + "@@@STEP_LOG_LINE@json.output@{@@@", + "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@", + "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"40-chars-fake-of-the-package-instance_id\", @@@", + "@@@STEP_LOG_LINE@json.output@ \"package\": \"gn/gn/${platform}\"@@@", + "@@@STEP_LOG_LINE@json.output@ }@@@", + "@@@STEP_LOG_LINE@json.output@}@@@", + "@@@STEP_LOG_END@json.output@@@" + ] + }, + { + "cmd": [ + "git", + "rev-parse", + "HEAD" + ], + "cwd": "[START_DIR]/gn", + "name": "rev-parse", + "stdout": "/path/to/tmp/" + }, + { + "cmd": [ + "cipd", + "search", + "gn/gn/${platform}", + "-tag", + "git_revision:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "-json-output", + "/path/to/tmp/json" + ], + "name": "cipd search gn/gn/${platform} git_revision:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "~followup_annotations": [ + "@@@STEP_LOG_LINE@json.output@{@@@", + "@@@STEP_LOG_LINE@json.output@ \"result\": [@@@", + "@@@STEP_LOG_LINE@json.output@ {@@@", + "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"resolved-instance_id-of-git_revision:aaa\", @@@", + "@@@STEP_LOG_LINE@json.output@ \"package\": \"gn/gn/${platform}\"@@@", + "@@@STEP_LOG_LINE@json.output@ }@@@", + "@@@STEP_LOG_LINE@json.output@ ]@@@", + "@@@STEP_LOG_LINE@json.output@}@@@", + "@@@STEP_LOG_END@json.output@@@" + ] + }, + { + "cmd": [], + "name": "Package is up-to-date" + }, + { "name": "$result", "recipe_result": null, "status_code": 0
diff --git a/infra/recipes/gn.expected/cq_linux.json b/infra/recipes/gn.expected/cipd_register.json similarity index 67% copy from infra/recipes/gn.expected/cq_linux.json copy to infra/recipes/gn.expected/cipd_register.json index b974d7b..dd37807 100644 --- a/infra/recipes/gn.expected/cq_linux.json +++ b/infra/recipes/gn.expected/cipd_register.json
@@ -20,7 +20,7 @@ "git", "fetch", "https://gn.googlesource.com/gn", - "refs/heads/master" + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" ], "cwd": "[START_DIR]/gn", "infra_step": true, @@ -44,33 +44,6 @@ }, { "cmd": [ - "git", - "fetch", - "https://gn-review.googlesource.com/gn", - "refs/changes/00/1000/1" - ], - "cwd": "[START_DIR]/gn", - "infra_step": true, - "name": "git.fetch 1000/1", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "git", - "cherry-pick", - "FETCH_HEAD" - ], - "cwd": "[START_DIR]/gn", - "infra_step": true, - "name": "git.cherry-pick 1000/1", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ "cipd", "ensure", "-root", @@ -214,6 +187,81 @@ ] }, { + "cmd": [ + "cipd", + "pkg-build", + "-pkg-def", + "{\"data\": [{\"file\": \"gn\"}, {\"version_file\": \".versions/gn.cipd_version\"}], \"install_mode\": \"copy\", \"package\": \"gn/gn/${platform}\", \"root\": \"[START_DIR]/gn/out\"}", + "-out", + "[CLEANUP]/gn.cipd", + "-json-output", + "/path/to/tmp/json" + ], + "name": "build gn/gn/${platform}", + "~followup_annotations": [ + "@@@STEP_LOG_LINE@json.output@{@@@", + "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@", + "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"40-chars-fake-of-the-package-instance_id\", @@@", + "@@@STEP_LOG_LINE@json.output@ \"package\": \"gn/gn/${platform}\"@@@", + "@@@STEP_LOG_LINE@json.output@ }@@@", + "@@@STEP_LOG_LINE@json.output@}@@@", + "@@@STEP_LOG_END@json.output@@@" + ] + }, + { + "cmd": [ + "git", + "rev-parse", + "HEAD" + ], + "cwd": "[START_DIR]/gn", + "name": "rev-parse", + "stdout": "/path/to/tmp/" + }, + { + "cmd": [ + "cipd", + "search", + "gn/gn/${platform}", + "-tag", + "git_revision:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "-json-output", + "/path/to/tmp/json" + ], + "name": "cipd search gn/gn/${platform} git_revision:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "~followup_annotations": [ + "@@@STEP_LOG_LINE@json.output@{@@@", + "@@@STEP_LOG_LINE@json.output@ \"result\": []@@@", + "@@@STEP_LOG_LINE@json.output@}@@@", + "@@@STEP_LOG_END@json.output@@@" + ] + }, + { + "cmd": [ + "cipd", + "pkg-register", + "[CLEANUP]/gn.cipd", + "-ref", + "latest", + "-tag", + "git_repository:https://gn.googlesource.com/gn", + "-tag", + "git_revision:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "-json-output", + "/path/to/tmp/json" + ], + "name": "register gn/gn/${platform}", + "~followup_annotations": [ + "@@@STEP_LOG_LINE@json.output@{@@@", + "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@", + "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"40-chars-fake-of-the-package-instance_id\", @@@", + "@@@STEP_LOG_LINE@json.output@ \"package\": \"gn/gn/${platform}\"@@@", + "@@@STEP_LOG_LINE@json.output@ }@@@", + "@@@STEP_LOG_LINE@json.output@}@@@", + "@@@STEP_LOG_END@json.output@@@" + ] + }, + { "name": "$result", "recipe_result": null, "status_code": 0
diff --git a/infra/recipes/gn.expected/cq_win.json b/infra/recipes/gn.expected/cq.json similarity index 99% rename from infra/recipes/gn.expected/cq_win.json rename to infra/recipes/gn.expected/cq.json index 7e56381..71cb320 100644 --- a/infra/recipes/gn.expected/cq_win.json +++ b/infra/recipes/gn.expected/cq.json
@@ -46,7 +46,7 @@ "cmd": [ "git", "fetch", - "https://gn-review.googlesource.com/gn", + "https://gn.googlesource.com/gn", "refs/changes/00/1000/1" ], "cwd": "[START_DIR]\\gn",
diff --git a/infra/recipes/gn.expected/cq_mac.json b/infra/recipes/gn.expected/cq_mac.json deleted file mode 100644 index 6824324..0000000 --- a/infra/recipes/gn.expected/cq_mac.json +++ /dev/null
@@ -1,193 +0,0 @@ -[ - { - "cmd": [], - "name": "git" - }, - { - "cmd": [ - "git", - "init", - "[START_DIR]/gn" - ], - "infra_step": true, - "name": "git.init", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "git", - "fetch", - "https://gn.googlesource.com/gn", - "refs/heads/master" - ], - "cwd": "[START_DIR]/gn", - "infra_step": true, - "name": "git.fetch", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "git", - "checkout", - "FETCH_HEAD" - ], - "cwd": "[START_DIR]/gn", - "infra_step": true, - "name": "git.checkout", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "git", - "fetch", - "https://gn-review.googlesource.com/gn", - "refs/changes/00/1000/1" - ], - "cwd": "[START_DIR]/gn", - "infra_step": true, - "name": "git.fetch 1000/1", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "git", - "cherry-pick", - "FETCH_HEAD" - ], - "cwd": "[START_DIR]/gn", - "infra_step": true, - "name": "git.cherry-pick 1000/1", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "cipd", - "ensure", - "-root", - "[START_DIR]/cipd", - "-ensure-file", - "infra/ninja/${platform} version:1.8.2", - "-json-output", - "/path/to/tmp/json" - ], - "infra_step": true, - "name": "ensure_installed", - "~followup_annotations": [ - "@@@STEP_LOG_LINE@json.output@{@@@", - "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@", - "@@@STEP_LOG_LINE@json.output@ \"\": [@@@", - "@@@STEP_LOG_LINE@json.output@ {@@@", - "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"resolved-instance_id-of-version:1.8.2---\", @@@", - "@@@STEP_LOG_LINE@json.output@ \"package\": \"infra/ninja/${platform}\"@@@", - "@@@STEP_LOG_LINE@json.output@ }@@@", - "@@@STEP_LOG_LINE@json.output@ ]@@@", - "@@@STEP_LOG_LINE@json.output@ }@@@", - "@@@STEP_LOG_LINE@json.output@}@@@", - "@@@STEP_LOG_END@json.output@@@" - ] - }, - { - "cmd": [], - "name": "debug" - }, - { - "cmd": [], - "name": "debug.build", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "python", - "-u", - "[START_DIR]/gn/build/gen.py", - "-d" - ], - "cwd": "[START_DIR]/gn", - "name": "debug.build.generate", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@2@@@" - ] - }, - { - "cmd": [ - "[START_DIR]/cipd/ninja", - "-C", - "[START_DIR]/gn/out" - ], - "cwd": "[START_DIR]/gn", - "name": "debug.build.ninja", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@2@@@" - ] - }, - { - "cmd": [ - "[START_DIR]/gn/out/gn_unittests" - ], - "name": "debug.test", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [], - "name": "release" - }, - { - "cmd": [], - "name": "release.build", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "cmd": [ - "python", - "-u", - "[START_DIR]/gn/build/gen.py" - ], - "cwd": "[START_DIR]/gn", - "name": "release.build.generate", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@2@@@" - ] - }, - { - "cmd": [ - "[START_DIR]/cipd/ninja", - "-C", - "[START_DIR]/gn/out" - ], - "cwd": "[START_DIR]/gn", - "name": "release.build.ninja", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@2@@@" - ] - }, - { - "cmd": [ - "[START_DIR]/gn/out/gn_unittests" - ], - "name": "release.test", - "~followup_annotations": [ - "@@@STEP_NEST_LEVEL@1@@@" - ] - }, - { - "name": "$result", - "recipe_result": null, - "status_code": 0 - } -] \ No newline at end of file
diff --git a/infra/recipes/gn.py b/infra/recipes/gn.py index d0a195c..2048936 100644 --- a/infra/recipes/gn.py +++ b/infra/recipes/gn.py
@@ -3,21 +3,29 @@ # that can be found in the LICENSE file. """Recipe for building GN.""" +from recipe_engine.recipe_api import Property + DEPS = [ 'recipe_engine/buildbucket', 'recipe_engine/cipd', 'recipe_engine/context', + 'recipe_engine/file', 'recipe_engine/json', 'recipe_engine/path', 'recipe_engine/platform', 'recipe_engine/properties', 'recipe_engine/python', + 'recipe_engine/raw_io', 'recipe_engine/step', 'windows_sdk', ] +PROPERTIES = { + 'repository': Property(kind=str, default='https://gn.googlesource.com/gn'), +} -def RunSteps(api): + +def RunSteps(api, repository): src_dir = api.path['start_dir'].join('gn') with api.step.nest('git'), api.context(infra_steps=True): @@ -28,12 +36,11 @@ ref = ( build_input.gitiles_commit.id if build_input.gitiles_commit else 'refs/heads/master') - api.step('fetch', ['git', 'fetch', 'https://gn.googlesource.com/gn', ref]) + api.step('fetch', ['git', 'fetch', repository, ref]) api.step('checkout', ['git', 'checkout', 'FETCH_HEAD']) for change in build_input.gerrit_changes: api.step('fetch %s/%s' % (change.change, change.patchset), [ - 'git', 'fetch', - 'https://%s/gn' % change.host, + 'git', 'fetch', repository, 'refs/changes/%s/%s/%s' % (str(change.change)[-2:], change.change, change.patchset) ]) @@ -65,6 +72,7 @@ 'win': {}, }[api.platform.name] + # The order is important since release build will get uploaded to CIPD. configs = [ { 'name': 'debug', @@ -89,25 +97,75 @@ api.step('test', [src_dir.join('out', 'gn_unittests')]) + if build_input.gerrit_changes: + return + + cipd_pkg_name = 'gn/gn/${platform}' + gn = 'gn' + ('.exe' if api.platform.is_win else '') + + pkg_def = api.cipd.PackageDefinition( + package_name=cipd_pkg_name, + package_root=src_dir.join('out'), + install_mode='copy') + pkg_def.add_file(src_dir.join('out', gn)) + pkg_def.add_version_file('.versions/%s.cipd_version' % gn) + + cipd_pkg_file = api.path['cleanup'].join('gn.cipd') + + api.cipd.build_from_pkg( + pkg_def=pkg_def, + output_package=cipd_pkg_file, + ) + + if api.buildbucket.builder_id.project == 'infra-internal': + with api.context(cwd=src_dir): + revision = api.step( + 'rev-parse', ['git', 'rev-parse', 'HEAD'], + stdout=api.raw_io.output()).stdout.strip() + + cipd_pin = api.cipd.search(cipd_pkg_name, 'git_revision:' + revision) + if cipd_pin: + api.step('Package is up-to-date', cmd=None) + return + + api.cipd.register( + package_name=cipd_pkg_name, + package_path=cipd_pkg_file, + refs=['latest'], + tags={ + 'git_repository': repository, + 'git_revision': revision, + }, + ) + def GenTests(api): - for platform in ('linux', 'mac', 'win'): - yield (api.test('ci_' + platform) + api.platform.name(platform) + - api.properties(buildbucket={ - 'build': { - 'tags': [ - 'buildset:commit/gitiles/gn.googlesource.com/gn/+/' - 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', - ] - } - })) + REVISION = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' - yield ( - api.test('cq_' + platform) + api.platform.name(platform) + - api.properties(buildbucket={ - 'build': { - 'tags': [ - 'buildset:patch/gerrit/gn-review.googlesource.com/1000/1', - ] - } - })) + yield (api.test('ci') + api.platform.name('win') + api.buildbucket.ci_build( + git_repo='gn.googlesource.com/gn', + revision=REVISION, + )) + + yield (api.test('cq') + api.platform.name('win') + api.buildbucket.try_build( + gerrit_host='gn-review.googlesource.com', + change_number=1000, + patch_set=1, + )) + + yield (api.test('cipd_exists') + api.buildbucket.ci_build( + project='infra-internal', + git_repo='gn.googlesource.com/gn', + revision=REVISION, + ) + api.step_data('rev-parse', api.raw_io.stream_output(REVISION)) + + api.step_data('cipd search gn/gn/${platform} git_revision:' + REVISION, + api.cipd.example_search('gn/gn/${platform}', + ['git_revision:' + REVISION]))) + + yield (api.test('cipd_register') + api.buildbucket.ci_build( + project='infra-internal', + git_repo='gn.googlesource.com/gn', + revision=REVISION, + ) + api.step_data('rev-parse', api.raw_io.stream_output(REVISION)) + + api.step_data('cipd search gn/gn/${platform} git_revision:' + REVISION, + api.cipd.example_search('gn/gn/${platform}', [])))