Upload GN CIPD package
This change always builds the CIPD package for GN for CI builds and
when requested also uploads that package to CIPD server.
Change-Id: Idc62ed9c6b4c491426b2c7e26b87cad03a206d5e
Reviewed-on: https://gn-review.googlesource.com/1781
Reviewed-by: Andrii Shyshkalov <tandrii@google.com>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
Commit-Queue: Petr Hosek <phosek@google.com>
diff --git a/infra/README.recipes.md b/infra/README.recipes.md
index 9d447de..afa61e5 100644
--- a/infra/README.recipes.md
+++ b/infra/README.recipes.md
@@ -35,24 +35,26 @@
### *recipes* / [gn](/infra/recipes/gn.py)
-[DEPS](/infra/recipes/gn.py#6): [windows\_sdk](#recipe_modules-windows_sdk), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/step][recipe_engine/recipe_modules/step]
+[DEPS](/infra/recipes/gn.py#8): [windows\_sdk](#recipe_modules-windows_sdk), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step]
Recipe for building GN.
-— **def [RunSteps](/infra/recipes/gn.py#20)(api):**
+— **def [RunSteps](/infra/recipes/gn.py#29)(api, repository):**
### *recipes* / [windows\_sdk:examples/full](/infra/recipe_modules/windows_sdk/examples/full.py)
[DEPS](/infra/recipe_modules/windows_sdk/examples/full.py#5): [windows\_sdk](#recipe_modules-windows_sdk), [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
— **def [RunSteps](/infra/recipe_modules/windows_sdk/examples/full.py#13)(api):**
-[recipe_engine/recipe_modules/buildbucket]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-buildbucket
-[recipe_engine/recipe_modules/cipd]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-cipd
-[recipe_engine/recipe_modules/context]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-context
-[recipe_engine/recipe_modules/json]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-json
-[recipe_engine/recipe_modules/path]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-path
-[recipe_engine/recipe_modules/platform]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-platform
-[recipe_engine/recipe_modules/properties]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-properties
-[recipe_engine/recipe_modules/python]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-python
-[recipe_engine/recipe_modules/step]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/README.recipes.md#recipe_modules-step
-[recipe_engine/wkt/RecipeApi]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/40699af88c76d65ef1b3d364de221611603e2dc6/recipe_engine/recipe_api.py#1006
+[recipe_engine/recipe_modules/buildbucket]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-buildbucket
+[recipe_engine/recipe_modules/cipd]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-cipd
+[recipe_engine/recipe_modules/context]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-context
+[recipe_engine/recipe_modules/file]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-file
+[recipe_engine/recipe_modules/json]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-json
+[recipe_engine/recipe_modules/path]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-path
+[recipe_engine/recipe_modules/platform]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-platform
+[recipe_engine/recipe_modules/properties]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-properties
+[recipe_engine/recipe_modules/python]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-python
+[recipe_engine/recipe_modules/raw_io]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-raw_io
+[recipe_engine/recipe_modules/step]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/README.recipes.md#recipe_modules-step
+[recipe_engine/wkt/RecipeApi]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/94bb0a758456a5df6891677a8cf2876c07ea5f0c/recipe_engine/recipe_api.py#1006
diff --git a/infra/config/recipes.cfg b/infra/config/recipes.cfg
index d0130f8..9d8e189 100644
--- a/infra/config/recipes.cfg
+++ b/infra/config/recipes.cfg
@@ -3,7 +3,7 @@
"deps": {
"recipe_engine": {
"branch": "master",
- "revision": "40699af88c76d65ef1b3d364de221611603e2dc6",
+ "revision": "94bb0a758456a5df6891677a8cf2876c07ea5f0c",
"url": "https://chromium.googlesource.com/infra/luci/recipes-py"
}
},
diff --git a/infra/recipes.py b/infra/recipes.py
index fe6589d..d726c68 100755
--- a/infra/recipes.py
+++ b/infra/recipes.py
@@ -107,7 +107,9 @@
raise MalformedRecipesCfg(ex.message, recipes_cfg_path)
-GIT = 'git.bat' if sys.platform.startswith(('win', 'cygwin')) else 'git'
+_BAT = '.bat' if sys.platform.startswith(('win', 'cygwin')) else ''
+GIT = 'git' + _BAT
+VPYTHON = 'vpython' + _BAT
def _subprocess_call(argv, **kwargs):
@@ -208,7 +210,7 @@
engine_path = checkout_engine(engine_override, repo_root, recipes_cfg_path)
return _subprocess_call([
- sys.executable, '-u',
+ VPYTHON, '-u',
os.path.join(engine_path, 'recipes.py')] + args)
diff --git a/infra/recipes/gn.expected/ci_win.json b/infra/recipes/gn.expected/ci.json
similarity index 91%
rename from infra/recipes/gn.expected/ci_win.json
rename to infra/recipes/gn.expected/ci.json
index 75e7300..d6ce20e 100644
--- a/infra/recipes/gn.expected/ci_win.json
+++ b/infra/recipes/gn.expected/ci.json
@@ -345,6 +345,28 @@
]
},
{
+ "cmd": [
+ "cipd.bat",
+ "pkg-build",
+ "-pkg-def",
+ "{\"data\": [{\"file\": \"gn.exe\"}, {\"version_file\": \".versions/gn.exe.cipd_version\"}], \"install_mode\": \"copy\", \"package\": \"gn/gn/${platform}\", \"root\": \"[START_DIR]\\\\gn\\\\out\"}",
+ "-out",
+ "[CLEANUP]\\gn.cipd",
+ "-json-output",
+ "/path/to/tmp/json"
+ ],
+ "name": "build gn/gn/${platform}",
+ "~followup_annotations": [
+ "@@@STEP_LOG_LINE@json.output@{@@@",
+ "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@",
+ "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"40-chars-fake-of-the-package-instance_id\", @@@",
+ "@@@STEP_LOG_LINE@json.output@ \"package\": \"gn/gn/${platform}\"@@@",
+ "@@@STEP_LOG_LINE@json.output@ }@@@",
+ "@@@STEP_LOG_LINE@json.output@}@@@",
+ "@@@STEP_LOG_END@json.output@@@"
+ ]
+ },
+ {
"name": "$result",
"recipe_result": null,
"status_code": 0
diff --git a/infra/recipes/gn.expected/ci_linux.json b/infra/recipes/gn.expected/ci_linux.json
deleted file mode 100644
index b313dc9..0000000
--- a/infra/recipes/gn.expected/ci_linux.json
+++ /dev/null
@@ -1,194 +0,0 @@
-[
- {
- "cmd": [],
- "name": "git"
- },
- {
- "cmd": [
- "git",
- "init",
- "[START_DIR]/gn"
- ],
- "infra_step": true,
- "name": "git.init",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "git",
- "fetch",
- "https://gn.googlesource.com/gn",
- "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
- ],
- "cwd": "[START_DIR]/gn",
- "infra_step": true,
- "name": "git.fetch",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "git",
- "checkout",
- "FETCH_HEAD"
- ],
- "cwd": "[START_DIR]/gn",
- "infra_step": true,
- "name": "git.checkout",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "cipd",
- "ensure",
- "-root",
- "[START_DIR]/cipd",
- "-ensure-file",
- "fuchsia/clang/${platform} goma\ninfra/ninja/${platform} version:1.8.2",
- "-json-output",
- "/path/to/tmp/json"
- ],
- "infra_step": true,
- "name": "ensure_installed",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@json.output@{@@@",
- "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@",
- "@@@STEP_LOG_LINE@json.output@ \"\": [@@@",
- "@@@STEP_LOG_LINE@json.output@ {@@@",
- "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"resolved-instance_id-of-goma------------\", @@@",
- "@@@STEP_LOG_LINE@json.output@ \"package\": \"fuchsia/clang/${platform}\"@@@",
- "@@@STEP_LOG_LINE@json.output@ }, @@@",
- "@@@STEP_LOG_LINE@json.output@ {@@@",
- "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"resolved-instance_id-of-version:1.8.2---\", @@@",
- "@@@STEP_LOG_LINE@json.output@ \"package\": \"infra/ninja/${platform}\"@@@",
- "@@@STEP_LOG_LINE@json.output@ }@@@",
- "@@@STEP_LOG_LINE@json.output@ ]@@@",
- "@@@STEP_LOG_LINE@json.output@ }@@@",
- "@@@STEP_LOG_LINE@json.output@}@@@",
- "@@@STEP_LOG_END@json.output@@@"
- ]
- },
- {
- "cmd": [],
- "name": "debug"
- },
- {
- "cmd": [],
- "name": "debug.build",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "python",
- "-u",
- "[START_DIR]/gn/build/gen.py",
- "-d"
- ],
- "cwd": "[START_DIR]/gn",
- "env": {
- "AR": "[START_DIR]/cipd/bin/llvm-ar",
- "CC": "[START_DIR]/cipd/bin/clang",
- "CXX": "[START_DIR]/cipd/bin/clang++",
- "LDFLAGS": "-static-libstdc++ -ldl -lpthread"
- },
- "name": "debug.build.generate",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@2@@@"
- ]
- },
- {
- "cmd": [
- "[START_DIR]/cipd/ninja",
- "-C",
- "[START_DIR]/gn/out"
- ],
- "cwd": "[START_DIR]/gn",
- "env": {
- "AR": "[START_DIR]/cipd/bin/llvm-ar",
- "CC": "[START_DIR]/cipd/bin/clang",
- "CXX": "[START_DIR]/cipd/bin/clang++",
- "LDFLAGS": "-static-libstdc++ -ldl -lpthread"
- },
- "name": "debug.build.ninja",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@2@@@"
- ]
- },
- {
- "cmd": [
- "[START_DIR]/gn/out/gn_unittests"
- ],
- "name": "debug.test",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [],
- "name": "release"
- },
- {
- "cmd": [],
- "name": "release.build",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "python",
- "-u",
- "[START_DIR]/gn/build/gen.py"
- ],
- "cwd": "[START_DIR]/gn",
- "env": {
- "AR": "[START_DIR]/cipd/bin/llvm-ar",
- "CC": "[START_DIR]/cipd/bin/clang",
- "CXX": "[START_DIR]/cipd/bin/clang++",
- "LDFLAGS": "-static-libstdc++ -ldl -lpthread"
- },
- "name": "release.build.generate",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@2@@@"
- ]
- },
- {
- "cmd": [
- "[START_DIR]/cipd/ninja",
- "-C",
- "[START_DIR]/gn/out"
- ],
- "cwd": "[START_DIR]/gn",
- "env": {
- "AR": "[START_DIR]/cipd/bin/llvm-ar",
- "CC": "[START_DIR]/cipd/bin/clang",
- "CXX": "[START_DIR]/cipd/bin/clang++",
- "LDFLAGS": "-static-libstdc++ -ldl -lpthread"
- },
- "name": "release.build.ninja",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@2@@@"
- ]
- },
- {
- "cmd": [
- "[START_DIR]/gn/out/gn_unittests"
- ],
- "name": "release.test",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "name": "$result",
- "recipe_result": null,
- "status_code": 0
- }
-]
\ No newline at end of file
diff --git a/infra/recipes/gn.expected/ci_mac.json b/infra/recipes/gn.expected/ci_mac.json
deleted file mode 100644
index c4f9847..0000000
--- a/infra/recipes/gn.expected/ci_mac.json
+++ /dev/null
@@ -1,166 +0,0 @@
-[
- {
- "cmd": [],
- "name": "git"
- },
- {
- "cmd": [
- "git",
- "init",
- "[START_DIR]/gn"
- ],
- "infra_step": true,
- "name": "git.init",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "git",
- "fetch",
- "https://gn.googlesource.com/gn",
- "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
- ],
- "cwd": "[START_DIR]/gn",
- "infra_step": true,
- "name": "git.fetch",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "git",
- "checkout",
- "FETCH_HEAD"
- ],
- "cwd": "[START_DIR]/gn",
- "infra_step": true,
- "name": "git.checkout",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "cipd",
- "ensure",
- "-root",
- "[START_DIR]/cipd",
- "-ensure-file",
- "infra/ninja/${platform} version:1.8.2",
- "-json-output",
- "/path/to/tmp/json"
- ],
- "infra_step": true,
- "name": "ensure_installed",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@json.output@{@@@",
- "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@",
- "@@@STEP_LOG_LINE@json.output@ \"\": [@@@",
- "@@@STEP_LOG_LINE@json.output@ {@@@",
- "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"resolved-instance_id-of-version:1.8.2---\", @@@",
- "@@@STEP_LOG_LINE@json.output@ \"package\": \"infra/ninja/${platform}\"@@@",
- "@@@STEP_LOG_LINE@json.output@ }@@@",
- "@@@STEP_LOG_LINE@json.output@ ]@@@",
- "@@@STEP_LOG_LINE@json.output@ }@@@",
- "@@@STEP_LOG_LINE@json.output@}@@@",
- "@@@STEP_LOG_END@json.output@@@"
- ]
- },
- {
- "cmd": [],
- "name": "debug"
- },
- {
- "cmd": [],
- "name": "debug.build",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "python",
- "-u",
- "[START_DIR]/gn/build/gen.py",
- "-d"
- ],
- "cwd": "[START_DIR]/gn",
- "name": "debug.build.generate",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@2@@@"
- ]
- },
- {
- "cmd": [
- "[START_DIR]/cipd/ninja",
- "-C",
- "[START_DIR]/gn/out"
- ],
- "cwd": "[START_DIR]/gn",
- "name": "debug.build.ninja",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@2@@@"
- ]
- },
- {
- "cmd": [
- "[START_DIR]/gn/out/gn_unittests"
- ],
- "name": "debug.test",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [],
- "name": "release"
- },
- {
- "cmd": [],
- "name": "release.build",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "python",
- "-u",
- "[START_DIR]/gn/build/gen.py"
- ],
- "cwd": "[START_DIR]/gn",
- "name": "release.build.generate",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@2@@@"
- ]
- },
- {
- "cmd": [
- "[START_DIR]/cipd/ninja",
- "-C",
- "[START_DIR]/gn/out"
- ],
- "cwd": "[START_DIR]/gn",
- "name": "release.build.ninja",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@2@@@"
- ]
- },
- {
- "cmd": [
- "[START_DIR]/gn/out/gn_unittests"
- ],
- "name": "release.test",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "name": "$result",
- "recipe_result": null,
- "status_code": 0
- }
-]
\ No newline at end of file
diff --git a/infra/recipes/gn.expected/cq_linux.json b/infra/recipes/gn.expected/cipd_exists.json
similarity index 71%
rename from infra/recipes/gn.expected/cq_linux.json
rename to infra/recipes/gn.expected/cipd_exists.json
index b974d7b..c63252d 100644
--- a/infra/recipes/gn.expected/cq_linux.json
+++ b/infra/recipes/gn.expected/cipd_exists.json
@@ -20,7 +20,7 @@
"git",
"fetch",
"https://gn.googlesource.com/gn",
- "refs/heads/master"
+ "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
],
"cwd": "[START_DIR]/gn",
"infra_step": true,
@@ -44,33 +44,6 @@
},
{
"cmd": [
- "git",
- "fetch",
- "https://gn-review.googlesource.com/gn",
- "refs/changes/00/1000/1"
- ],
- "cwd": "[START_DIR]/gn",
- "infra_step": true,
- "name": "git.fetch 1000/1",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "git",
- "cherry-pick",
- "FETCH_HEAD"
- ],
- "cwd": "[START_DIR]/gn",
- "infra_step": true,
- "name": "git.cherry-pick 1000/1",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
"cipd",
"ensure",
"-root",
@@ -214,6 +187,65 @@
]
},
{
+ "cmd": [
+ "cipd",
+ "pkg-build",
+ "-pkg-def",
+ "{\"data\": [{\"file\": \"gn\"}, {\"version_file\": \".versions/gn.cipd_version\"}], \"install_mode\": \"copy\", \"package\": \"gn/gn/${platform}\", \"root\": \"[START_DIR]/gn/out\"}",
+ "-out",
+ "[CLEANUP]/gn.cipd",
+ "-json-output",
+ "/path/to/tmp/json"
+ ],
+ "name": "build gn/gn/${platform}",
+ "~followup_annotations": [
+ "@@@STEP_LOG_LINE@json.output@{@@@",
+ "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@",
+ "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"40-chars-fake-of-the-package-instance_id\", @@@",
+ "@@@STEP_LOG_LINE@json.output@ \"package\": \"gn/gn/${platform}\"@@@",
+ "@@@STEP_LOG_LINE@json.output@ }@@@",
+ "@@@STEP_LOG_LINE@json.output@}@@@",
+ "@@@STEP_LOG_END@json.output@@@"
+ ]
+ },
+ {
+ "cmd": [
+ "git",
+ "rev-parse",
+ "HEAD"
+ ],
+ "cwd": "[START_DIR]/gn",
+ "name": "rev-parse",
+ "stdout": "/path/to/tmp/"
+ },
+ {
+ "cmd": [
+ "cipd",
+ "search",
+ "gn/gn/${platform}",
+ "-tag",
+ "git_revision:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+ "-json-output",
+ "/path/to/tmp/json"
+ ],
+ "name": "cipd search gn/gn/${platform} git_revision:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+ "~followup_annotations": [
+ "@@@STEP_LOG_LINE@json.output@{@@@",
+ "@@@STEP_LOG_LINE@json.output@ \"result\": [@@@",
+ "@@@STEP_LOG_LINE@json.output@ {@@@",
+ "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"resolved-instance_id-of-git_revision:aaa\", @@@",
+ "@@@STEP_LOG_LINE@json.output@ \"package\": \"gn/gn/${platform}\"@@@",
+ "@@@STEP_LOG_LINE@json.output@ }@@@",
+ "@@@STEP_LOG_LINE@json.output@ ]@@@",
+ "@@@STEP_LOG_LINE@json.output@}@@@",
+ "@@@STEP_LOG_END@json.output@@@"
+ ]
+ },
+ {
+ "cmd": [],
+ "name": "Package is up-to-date"
+ },
+ {
"name": "$result",
"recipe_result": null,
"status_code": 0
diff --git a/infra/recipes/gn.expected/cq_linux.json b/infra/recipes/gn.expected/cipd_register.json
similarity index 67%
copy from infra/recipes/gn.expected/cq_linux.json
copy to infra/recipes/gn.expected/cipd_register.json
index b974d7b..dd37807 100644
--- a/infra/recipes/gn.expected/cq_linux.json
+++ b/infra/recipes/gn.expected/cipd_register.json
@@ -20,7 +20,7 @@
"git",
"fetch",
"https://gn.googlesource.com/gn",
- "refs/heads/master"
+ "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
],
"cwd": "[START_DIR]/gn",
"infra_step": true,
@@ -44,33 +44,6 @@
},
{
"cmd": [
- "git",
- "fetch",
- "https://gn-review.googlesource.com/gn",
- "refs/changes/00/1000/1"
- ],
- "cwd": "[START_DIR]/gn",
- "infra_step": true,
- "name": "git.fetch 1000/1",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "git",
- "cherry-pick",
- "FETCH_HEAD"
- ],
- "cwd": "[START_DIR]/gn",
- "infra_step": true,
- "name": "git.cherry-pick 1000/1",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
"cipd",
"ensure",
"-root",
@@ -214,6 +187,81 @@
]
},
{
+ "cmd": [
+ "cipd",
+ "pkg-build",
+ "-pkg-def",
+ "{\"data\": [{\"file\": \"gn\"}, {\"version_file\": \".versions/gn.cipd_version\"}], \"install_mode\": \"copy\", \"package\": \"gn/gn/${platform}\", \"root\": \"[START_DIR]/gn/out\"}",
+ "-out",
+ "[CLEANUP]/gn.cipd",
+ "-json-output",
+ "/path/to/tmp/json"
+ ],
+ "name": "build gn/gn/${platform}",
+ "~followup_annotations": [
+ "@@@STEP_LOG_LINE@json.output@{@@@",
+ "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@",
+ "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"40-chars-fake-of-the-package-instance_id\", @@@",
+ "@@@STEP_LOG_LINE@json.output@ \"package\": \"gn/gn/${platform}\"@@@",
+ "@@@STEP_LOG_LINE@json.output@ }@@@",
+ "@@@STEP_LOG_LINE@json.output@}@@@",
+ "@@@STEP_LOG_END@json.output@@@"
+ ]
+ },
+ {
+ "cmd": [
+ "git",
+ "rev-parse",
+ "HEAD"
+ ],
+ "cwd": "[START_DIR]/gn",
+ "name": "rev-parse",
+ "stdout": "/path/to/tmp/"
+ },
+ {
+ "cmd": [
+ "cipd",
+ "search",
+ "gn/gn/${platform}",
+ "-tag",
+ "git_revision:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+ "-json-output",
+ "/path/to/tmp/json"
+ ],
+ "name": "cipd search gn/gn/${platform} git_revision:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+ "~followup_annotations": [
+ "@@@STEP_LOG_LINE@json.output@{@@@",
+ "@@@STEP_LOG_LINE@json.output@ \"result\": []@@@",
+ "@@@STEP_LOG_LINE@json.output@}@@@",
+ "@@@STEP_LOG_END@json.output@@@"
+ ]
+ },
+ {
+ "cmd": [
+ "cipd",
+ "pkg-register",
+ "[CLEANUP]/gn.cipd",
+ "-ref",
+ "latest",
+ "-tag",
+ "git_repository:https://gn.googlesource.com/gn",
+ "-tag",
+ "git_revision:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+ "-json-output",
+ "/path/to/tmp/json"
+ ],
+ "name": "register gn/gn/${platform}",
+ "~followup_annotations": [
+ "@@@STEP_LOG_LINE@json.output@{@@@",
+ "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@",
+ "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"40-chars-fake-of-the-package-instance_id\", @@@",
+ "@@@STEP_LOG_LINE@json.output@ \"package\": \"gn/gn/${platform}\"@@@",
+ "@@@STEP_LOG_LINE@json.output@ }@@@",
+ "@@@STEP_LOG_LINE@json.output@}@@@",
+ "@@@STEP_LOG_END@json.output@@@"
+ ]
+ },
+ {
"name": "$result",
"recipe_result": null,
"status_code": 0
diff --git a/infra/recipes/gn.expected/cq_win.json b/infra/recipes/gn.expected/cq.json
similarity index 99%
rename from infra/recipes/gn.expected/cq_win.json
rename to infra/recipes/gn.expected/cq.json
index 7e56381..71cb320 100644
--- a/infra/recipes/gn.expected/cq_win.json
+++ b/infra/recipes/gn.expected/cq.json
@@ -46,7 +46,7 @@
"cmd": [
"git",
"fetch",
- "https://gn-review.googlesource.com/gn",
+ "https://gn.googlesource.com/gn",
"refs/changes/00/1000/1"
],
"cwd": "[START_DIR]\\gn",
diff --git a/infra/recipes/gn.expected/cq_mac.json b/infra/recipes/gn.expected/cq_mac.json
deleted file mode 100644
index 6824324..0000000
--- a/infra/recipes/gn.expected/cq_mac.json
+++ /dev/null
@@ -1,193 +0,0 @@
-[
- {
- "cmd": [],
- "name": "git"
- },
- {
- "cmd": [
- "git",
- "init",
- "[START_DIR]/gn"
- ],
- "infra_step": true,
- "name": "git.init",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "git",
- "fetch",
- "https://gn.googlesource.com/gn",
- "refs/heads/master"
- ],
- "cwd": "[START_DIR]/gn",
- "infra_step": true,
- "name": "git.fetch",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "git",
- "checkout",
- "FETCH_HEAD"
- ],
- "cwd": "[START_DIR]/gn",
- "infra_step": true,
- "name": "git.checkout",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "git",
- "fetch",
- "https://gn-review.googlesource.com/gn",
- "refs/changes/00/1000/1"
- ],
- "cwd": "[START_DIR]/gn",
- "infra_step": true,
- "name": "git.fetch 1000/1",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "git",
- "cherry-pick",
- "FETCH_HEAD"
- ],
- "cwd": "[START_DIR]/gn",
- "infra_step": true,
- "name": "git.cherry-pick 1000/1",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "cipd",
- "ensure",
- "-root",
- "[START_DIR]/cipd",
- "-ensure-file",
- "infra/ninja/${platform} version:1.8.2",
- "-json-output",
- "/path/to/tmp/json"
- ],
- "infra_step": true,
- "name": "ensure_installed",
- "~followup_annotations": [
- "@@@STEP_LOG_LINE@json.output@{@@@",
- "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@",
- "@@@STEP_LOG_LINE@json.output@ \"\": [@@@",
- "@@@STEP_LOG_LINE@json.output@ {@@@",
- "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"resolved-instance_id-of-version:1.8.2---\", @@@",
- "@@@STEP_LOG_LINE@json.output@ \"package\": \"infra/ninja/${platform}\"@@@",
- "@@@STEP_LOG_LINE@json.output@ }@@@",
- "@@@STEP_LOG_LINE@json.output@ ]@@@",
- "@@@STEP_LOG_LINE@json.output@ }@@@",
- "@@@STEP_LOG_LINE@json.output@}@@@",
- "@@@STEP_LOG_END@json.output@@@"
- ]
- },
- {
- "cmd": [],
- "name": "debug"
- },
- {
- "cmd": [],
- "name": "debug.build",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "python",
- "-u",
- "[START_DIR]/gn/build/gen.py",
- "-d"
- ],
- "cwd": "[START_DIR]/gn",
- "name": "debug.build.generate",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@2@@@"
- ]
- },
- {
- "cmd": [
- "[START_DIR]/cipd/ninja",
- "-C",
- "[START_DIR]/gn/out"
- ],
- "cwd": "[START_DIR]/gn",
- "name": "debug.build.ninja",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@2@@@"
- ]
- },
- {
- "cmd": [
- "[START_DIR]/gn/out/gn_unittests"
- ],
- "name": "debug.test",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [],
- "name": "release"
- },
- {
- "cmd": [],
- "name": "release.build",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "cmd": [
- "python",
- "-u",
- "[START_DIR]/gn/build/gen.py"
- ],
- "cwd": "[START_DIR]/gn",
- "name": "release.build.generate",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@2@@@"
- ]
- },
- {
- "cmd": [
- "[START_DIR]/cipd/ninja",
- "-C",
- "[START_DIR]/gn/out"
- ],
- "cwd": "[START_DIR]/gn",
- "name": "release.build.ninja",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@2@@@"
- ]
- },
- {
- "cmd": [
- "[START_DIR]/gn/out/gn_unittests"
- ],
- "name": "release.test",
- "~followup_annotations": [
- "@@@STEP_NEST_LEVEL@1@@@"
- ]
- },
- {
- "name": "$result",
- "recipe_result": null,
- "status_code": 0
- }
-]
\ No newline at end of file
diff --git a/infra/recipes/gn.py b/infra/recipes/gn.py
index d0a195c..2048936 100644
--- a/infra/recipes/gn.py
+++ b/infra/recipes/gn.py
@@ -3,21 +3,29 @@
# that can be found in the LICENSE file.
"""Recipe for building GN."""
+from recipe_engine.recipe_api import Property
+
DEPS = [
'recipe_engine/buildbucket',
'recipe_engine/cipd',
'recipe_engine/context',
+ 'recipe_engine/file',
'recipe_engine/json',
'recipe_engine/path',
'recipe_engine/platform',
'recipe_engine/properties',
'recipe_engine/python',
+ 'recipe_engine/raw_io',
'recipe_engine/step',
'windows_sdk',
]
+PROPERTIES = {
+ 'repository': Property(kind=str, default='https://gn.googlesource.com/gn'),
+}
-def RunSteps(api):
+
+def RunSteps(api, repository):
src_dir = api.path['start_dir'].join('gn')
with api.step.nest('git'), api.context(infra_steps=True):
@@ -28,12 +36,11 @@
ref = (
build_input.gitiles_commit.id
if build_input.gitiles_commit else 'refs/heads/master')
- api.step('fetch', ['git', 'fetch', 'https://gn.googlesource.com/gn', ref])
+ api.step('fetch', ['git', 'fetch', repository, ref])
api.step('checkout', ['git', 'checkout', 'FETCH_HEAD'])
for change in build_input.gerrit_changes:
api.step('fetch %s/%s' % (change.change, change.patchset), [
- 'git', 'fetch',
- 'https://%s/gn' % change.host,
+ 'git', 'fetch', repository,
'refs/changes/%s/%s/%s' %
(str(change.change)[-2:], change.change, change.patchset)
])
@@ -65,6 +72,7 @@
'win': {},
}[api.platform.name]
+ # The order is important since release build will get uploaded to CIPD.
configs = [
{
'name': 'debug',
@@ -89,25 +97,75 @@
api.step('test', [src_dir.join('out', 'gn_unittests')])
+ if build_input.gerrit_changes:
+ return
+
+ cipd_pkg_name = 'gn/gn/${platform}'
+ gn = 'gn' + ('.exe' if api.platform.is_win else '')
+
+ pkg_def = api.cipd.PackageDefinition(
+ package_name=cipd_pkg_name,
+ package_root=src_dir.join('out'),
+ install_mode='copy')
+ pkg_def.add_file(src_dir.join('out', gn))
+ pkg_def.add_version_file('.versions/%s.cipd_version' % gn)
+
+ cipd_pkg_file = api.path['cleanup'].join('gn.cipd')
+
+ api.cipd.build_from_pkg(
+ pkg_def=pkg_def,
+ output_package=cipd_pkg_file,
+ )
+
+ if api.buildbucket.builder_id.project == 'infra-internal':
+ with api.context(cwd=src_dir):
+ revision = api.step(
+ 'rev-parse', ['git', 'rev-parse', 'HEAD'],
+ stdout=api.raw_io.output()).stdout.strip()
+
+ cipd_pin = api.cipd.search(cipd_pkg_name, 'git_revision:' + revision)
+ if cipd_pin:
+ api.step('Package is up-to-date', cmd=None)
+ return
+
+ api.cipd.register(
+ package_name=cipd_pkg_name,
+ package_path=cipd_pkg_file,
+ refs=['latest'],
+ tags={
+ 'git_repository': repository,
+ 'git_revision': revision,
+ },
+ )
+
def GenTests(api):
- for platform in ('linux', 'mac', 'win'):
- yield (api.test('ci_' + platform) + api.platform.name(platform) +
- api.properties(buildbucket={
- 'build': {
- 'tags': [
- 'buildset:commit/gitiles/gn.googlesource.com/gn/+/'
- 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
- ]
- }
- }))
+ REVISION = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
- yield (
- api.test('cq_' + platform) + api.platform.name(platform) +
- api.properties(buildbucket={
- 'build': {
- 'tags': [
- 'buildset:patch/gerrit/gn-review.googlesource.com/1000/1',
- ]
- }
- }))
+ yield (api.test('ci') + api.platform.name('win') + api.buildbucket.ci_build(
+ git_repo='gn.googlesource.com/gn',
+ revision=REVISION,
+ ))
+
+ yield (api.test('cq') + api.platform.name('win') + api.buildbucket.try_build(
+ gerrit_host='gn-review.googlesource.com',
+ change_number=1000,
+ patch_set=1,
+ ))
+
+ yield (api.test('cipd_exists') + api.buildbucket.ci_build(
+ project='infra-internal',
+ git_repo='gn.googlesource.com/gn',
+ revision=REVISION,
+ ) + api.step_data('rev-parse', api.raw_io.stream_output(REVISION)) +
+ api.step_data('cipd search gn/gn/${platform} git_revision:' + REVISION,
+ api.cipd.example_search('gn/gn/${platform}',
+ ['git_revision:' + REVISION])))
+
+ yield (api.test('cipd_register') + api.buildbucket.ci_build(
+ project='infra-internal',
+ git_repo='gn.googlesource.com/gn',
+ revision=REVISION,
+ ) + api.step_data('rev-parse', api.raw_io.stream_output(REVISION)) +
+ api.step_data('cipd search gn/gn/${platform} git_revision:' + REVISION,
+ api.cipd.example_search('gn/gn/${platform}', [])))
