upload gn binary to CAS from CQ Currently, gn binary available in CIPD and gn binary built from my local machine has some performance difference (gn in CIPD is faster). And that makes local performance comparison is bit unreliable. In some cases, I'd like to compare performance of gn binary built by existing recipe pipeline instead of replicating them in local machine. For such purpose, this CL changes recipe to upload gn binary to CAS from CQ. And we can use uploaded binary from CQ for performance comparison without submitting CLs. I added project-gn-try-task-accounts to chromium-swarm-cas-read-write in https://chrome-infra-auth.appspot.com/auth/change_log?auth_db_rev=44767 for this, and confirmed the recipe behavir with $ led get-build 8818119718319643841 | led edit-recipe-bundle |\ led launch https://luci-milo.appspot.com/raw/build/logs.chromium.org/gn/led/tikuta_google.com/9a9e553c22da28cace97964ec59a5a2139451542efac5c00f0d9711b04584198/+/build.proto Bug: chromium:1062263 Change-Id: I676358f4715128d06027b673ba5144b68fa8212c Reviewed-on: https://gn-review.googlesource.com/c/gn/+/13401 Reviewed-by: Petr Hosek <phosek@google.com> Commit-Queue: Takuto Ikuta <tikuta@google.com>
diff --git a/infra/README.recipes.md b/infra/README.recipes.md index 20e5a6b..5a6ea74 100644 --- a/infra/README.recipes.md +++ b/infra/README.recipes.md
@@ -95,13 +95,13 @@ ### *recipes* / [gn](/infra/recipes/gn.py) -[DEPS](/infra/recipes/gn.py#10): [macos\_sdk](#recipe_modules-macos_sdk), [target](#recipe_modules-target), [windows\_sdk](#recipe_modules-windows_sdk), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step] +[DEPS](/infra/recipes/gn.py#10): [macos\_sdk](#recipe_modules-macos_sdk), [target](#recipe_modules-target), [windows\_sdk](#recipe_modules-windows_sdk), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/cas][recipe_engine/recipe_modules/cas], [recipe\_engine/cipd][recipe_engine/recipe_modules/cipd], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/json][recipe_engine/recipe_modules/json], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/python][recipe_engine/recipe_modules/python], [recipe\_engine/raw\_io][recipe_engine/recipe_modules/raw_io], [recipe\_engine/step][recipe_engine/recipe_modules/step] PYTHON_VERSION_COMPATIBILITY: PY3 Recipe for building GN. -— **def [RunSteps](/infra/recipes/gn.py#104)(api, repository):** +— **def [RunSteps](/infra/recipes/gn.py#105)(api, repository):** ### *recipes* / [macos\_sdk:examples/full](/infra/recipe_modules/macos_sdk/examples/full.py) [DEPS](/infra/recipe_modules/macos_sdk/examples/full.py#7): [macos\_sdk](#recipe_modules-macos_sdk), [recipe\_engine/platform][recipe_engine/recipe_modules/platform], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step] @@ -125,6 +125,7 @@ — **def [RunSteps](/infra/recipe_modules/windows_sdk/examples/full.py#15)(api):** [recipe_engine/recipe_modules/buildbucket]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/1b1ecd03e0b00399784c43add1465f685b6d1ab9/README.recipes.md#recipe_modules-buildbucket +[recipe_engine/recipe_modules/cas]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/1b1ecd03e0b00399784c43add1465f685b6d1ab9/README.recipes.md#recipe_modules-cas [recipe_engine/recipe_modules/cipd]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/1b1ecd03e0b00399784c43add1465f685b6d1ab9/README.recipes.md#recipe_modules-cipd [recipe_engine/recipe_modules/context]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/1b1ecd03e0b00399784c43add1465f685b6d1ab9/README.recipes.md#recipe_modules-context [recipe_engine/recipe_modules/file]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/1b1ecd03e0b00399784c43add1465f685b6d1ab9/README.recipes.md#recipe_modules-file
diff --git a/infra/recipes/gn.expected/cq_linux.json b/infra/recipes/gn.expected/cq_linux.json index d19691e..574d790 100644 --- a/infra/recipes/gn.expected/cq_linux.json +++ b/infra/recipes/gn.expected/cq_linux.json
@@ -703,6 +703,185 @@ }, { "cmd": [], + "name": "release.linux-amd64.upload", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@2@@@" + ] + }, + { + "cmd": [ + "vpython", + "-u", + "RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py", + "--json-output", + "/path/to/tmp/json", + "copy", + "RECIPE_MODULE[recipe_engine::cas]/resources/infra.sha1", + "/path/to/tmp/" + ], + "cwd": "[START_DIR]/gn", + "env": { + "AR": "[START_DIR]/cipd/bin/llvm-ar", + "CC": "[START_DIR]/cipd/bin/clang", + "CFLAGS": "--target=x86_64-linux-gnu --sysroot=[START_DIR]/cipd/sysroot", + "CXX": "[START_DIR]/cipd/bin/clang++", + "LDFLAGS": "--target=x86_64-linux-gnu --sysroot=[START_DIR]/cipd/sysroot -static-libstdc++" + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.linux-amd64.upload.read infra revision", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@3@@@", + "@@@STEP_LOG_LINE@infra.sha1@git_revision:mock_infra_git_revision@@@", + "@@@STEP_LOG_END@infra.sha1@@@" + ] + }, + { + "cmd": [], + "name": "release.linux-amd64.upload.install infra/tools/luci/cas", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@3@@@" + ] + }, + { + "cmd": [ + "vpython", + "-u", + "RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py", + "--json-output", + "/path/to/tmp/json", + "ensure-directory", + "--mode", + "0777", + "[START_DIR]/cipd_tool/infra/tools/luci/cas/git_revision%3Amock_infra_git_revision" + ], + "cwd": "[START_DIR]/gn", + "env": { + "AR": "[START_DIR]/cipd/bin/llvm-ar", + "CC": "[START_DIR]/cipd/bin/clang", + "CFLAGS": "--target=x86_64-linux-gnu --sysroot=[START_DIR]/cipd/sysroot", + "CXX": "[START_DIR]/cipd/bin/clang++", + "LDFLAGS": "--target=x86_64-linux-gnu --sysroot=[START_DIR]/cipd/sysroot -static-libstdc++" + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.linux-amd64.upload.install infra/tools/luci/cas.ensure package directory", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@4@@@" + ] + }, + { + "cmd": [ + "cipd", + "ensure", + "-root", + "[START_DIR]/cipd_tool/infra/tools/luci/cas/git_revision%3Amock_infra_git_revision", + "-ensure-file", + "infra/tools/luci/cas/${platform} git_revision:mock_infra_git_revision", + "-max-threads", + "0", + "-json-output", + "/path/to/tmp/json" + ], + "cwd": "[START_DIR]/gn", + "env": { + "AR": "[START_DIR]/cipd/bin/llvm-ar", + "CC": "[START_DIR]/cipd/bin/clang", + "CFLAGS": "--target=x86_64-linux-gnu --sysroot=[START_DIR]/cipd/sysroot", + "CXX": "[START_DIR]/cipd/bin/clang++", + "LDFLAGS": "--target=x86_64-linux-gnu --sysroot=[START_DIR]/cipd/sysroot -static-libstdc++" + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.linux-amd64.upload.install infra/tools/luci/cas.ensure_installed", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@4@@@", + "@@@STEP_LOG_LINE@json.output@{@@@", + "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@", + "@@@STEP_LOG_LINE@json.output@ \"\": [@@@", + "@@@STEP_LOG_LINE@json.output@ {@@@", + "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"resolved-instance_id-of-git_revision:moc\", @@@", + "@@@STEP_LOG_LINE@json.output@ \"package\": \"infra/tools/luci/cas/resolved-platform\"@@@", + "@@@STEP_LOG_LINE@json.output@ }@@@", + "@@@STEP_LOG_LINE@json.output@ ]@@@", + "@@@STEP_LOG_LINE@json.output@ }@@@", + "@@@STEP_LOG_LINE@json.output@}@@@", + "@@@STEP_LOG_END@json.output@@@" + ] + }, + { + "cmd": [ + "[START_DIR]/cipd_tool/infra/tools/luci/cas/git_revision%3Amock_infra_git_revision/cas", + "archive", + "-cas-instance", + "projects/example-cas-server/instances/default_instance", + "-dump-digest", + "/path/to/tmp/", + "-paths", + "[START_DIR]/gn/out:gn" + ], + "cwd": "[START_DIR]/gn", + "env": { + "AR": "[START_DIR]/cipd/bin/llvm-ar", + "CC": "[START_DIR]/cipd/bin/clang", + "CFLAGS": "--target=x86_64-linux-gnu --sysroot=[START_DIR]/cipd/sysroot", + "CXX": "[START_DIR]/cipd/bin/clang++", + "LDFLAGS": "--target=x86_64-linux-gnu --sysroot=[START_DIR]/cipd/sysroot -static-libstdc++" + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.linux-amd64.upload.upload binary to CAS", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@3@@@", + "@@@STEP_LINK@CAS UI@https://cas-viewer.appspot.com/projects/example-cas-server/instances/default_instance/blobs/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/0/tree@@@" + ] + }, + { + "cmd": [], "name": "release.linux-arm64", "~followup_annotations": [ "@@@STEP_NEST_LEVEL@1@@@" @@ -774,6 +953,51 @@ ] }, { + "cmd": [], + "name": "release.linux-arm64.upload", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@2@@@" + ] + }, + { + "cmd": [ + "[START_DIR]/cipd_tool/infra/tools/luci/cas/git_revision%3Amock_infra_git_revision/cas", + "archive", + "-cas-instance", + "projects/example-cas-server/instances/default_instance", + "-dump-digest", + "/path/to/tmp/", + "-paths", + "[START_DIR]/gn/out:gn" + ], + "cwd": "[START_DIR]/gn", + "env": { + "AR": "[START_DIR]/cipd/bin/llvm-ar", + "CC": "[START_DIR]/cipd/bin/clang", + "CFLAGS": "--target=aarch64-linux-gnu --sysroot=[START_DIR]/cipd/sysroot", + "CXX": "[START_DIR]/cipd/bin/clang++", + "LDFLAGS": "--target=aarch64-linux-gnu --sysroot=[START_DIR]/cipd/sysroot -static-libstdc++" + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.linux-arm64.upload.upload binary to CAS", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@3@@@", + "@@@STEP_LINK@CAS UI@https://cas-viewer.appspot.com/projects/example-cas-server/instances/default_instance/blobs/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/0/tree@@@" + ] + }, + { "name": "$result" } ] \ No newline at end of file
diff --git a/infra/recipes/gn.expected/cq_mac.json b/infra/recipes/gn.expected/cq_mac.json index 4017d0b..f9d745e 100644 --- a/infra/recipes/gn.expected/cq_mac.json +++ b/infra/recipes/gn.expected/cq_mac.json
@@ -605,6 +605,185 @@ ] }, { + "cmd": [], + "name": "release.mac-amd64.upload", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@2@@@" + ] + }, + { + "cmd": [ + "vpython", + "-u", + "RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py", + "--json-output", + "/path/to/tmp/json", + "copy", + "RECIPE_MODULE[recipe_engine::cas]/resources/infra.sha1", + "/path/to/tmp/" + ], + "cwd": "[START_DIR]/gn", + "env": { + "AR": "[START_DIR]/cipd/bin/llvm-ar", + "CC": "[START_DIR]/cipd/bin/clang", + "CFLAGS": "--target=x86_64-apple-darwin --sysroot=/some/xcode/path -nostdinc++ -cxx-isystem [CACHE]/macos_sdk/XCode.app/include/c++/v1", + "CXX": "[START_DIR]/cipd/bin/clang++", + "LDFLAGS": "--target=x86_64-apple-darwin --sysroot=/some/xcode/path" + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.mac-amd64.upload.read infra revision", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@3@@@", + "@@@STEP_LOG_LINE@infra.sha1@git_revision:mock_infra_git_revision@@@", + "@@@STEP_LOG_END@infra.sha1@@@" + ] + }, + { + "cmd": [], + "name": "release.mac-amd64.upload.install infra/tools/luci/cas", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@3@@@" + ] + }, + { + "cmd": [ + "vpython", + "-u", + "RECIPE_MODULE[recipe_engine::file]/resources/fileutil.py", + "--json-output", + "/path/to/tmp/json", + "ensure-directory", + "--mode", + "0777", + "[START_DIR]/cipd_tool/infra/tools/luci/cas/git_revision%3Amock_infra_git_revision" + ], + "cwd": "[START_DIR]/gn", + "env": { + "AR": "[START_DIR]/cipd/bin/llvm-ar", + "CC": "[START_DIR]/cipd/bin/clang", + "CFLAGS": "--target=x86_64-apple-darwin --sysroot=/some/xcode/path -nostdinc++ -cxx-isystem [CACHE]/macos_sdk/XCode.app/include/c++/v1", + "CXX": "[START_DIR]/cipd/bin/clang++", + "LDFLAGS": "--target=x86_64-apple-darwin --sysroot=/some/xcode/path" + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.mac-amd64.upload.install infra/tools/luci/cas.ensure package directory", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@4@@@" + ] + }, + { + "cmd": [ + "cipd", + "ensure", + "-root", + "[START_DIR]/cipd_tool/infra/tools/luci/cas/git_revision%3Amock_infra_git_revision", + "-ensure-file", + "infra/tools/luci/cas/${platform} git_revision:mock_infra_git_revision", + "-max-threads", + "0", + "-json-output", + "/path/to/tmp/json" + ], + "cwd": "[START_DIR]/gn", + "env": { + "AR": "[START_DIR]/cipd/bin/llvm-ar", + "CC": "[START_DIR]/cipd/bin/clang", + "CFLAGS": "--target=x86_64-apple-darwin --sysroot=/some/xcode/path -nostdinc++ -cxx-isystem [CACHE]/macos_sdk/XCode.app/include/c++/v1", + "CXX": "[START_DIR]/cipd/bin/clang++", + "LDFLAGS": "--target=x86_64-apple-darwin --sysroot=/some/xcode/path" + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.mac-amd64.upload.install infra/tools/luci/cas.ensure_installed", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@4@@@", + "@@@STEP_LOG_LINE@json.output@{@@@", + "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@", + "@@@STEP_LOG_LINE@json.output@ \"\": [@@@", + "@@@STEP_LOG_LINE@json.output@ {@@@", + "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"resolved-instance_id-of-git_revision:moc\", @@@", + "@@@STEP_LOG_LINE@json.output@ \"package\": \"infra/tools/luci/cas/resolved-platform\"@@@", + "@@@STEP_LOG_LINE@json.output@ }@@@", + "@@@STEP_LOG_LINE@json.output@ ]@@@", + "@@@STEP_LOG_LINE@json.output@ }@@@", + "@@@STEP_LOG_LINE@json.output@}@@@", + "@@@STEP_LOG_END@json.output@@@" + ] + }, + { + "cmd": [ + "[START_DIR]/cipd_tool/infra/tools/luci/cas/git_revision%3Amock_infra_git_revision/cas", + "archive", + "-cas-instance", + "projects/example-cas-server/instances/default_instance", + "-dump-digest", + "/path/to/tmp/", + "-paths", + "[START_DIR]/gn/out:gn" + ], + "cwd": "[START_DIR]/gn", + "env": { + "AR": "[START_DIR]/cipd/bin/llvm-ar", + "CC": "[START_DIR]/cipd/bin/clang", + "CFLAGS": "--target=x86_64-apple-darwin --sysroot=/some/xcode/path -nostdinc++ -cxx-isystem [CACHE]/macos_sdk/XCode.app/include/c++/v1", + "CXX": "[START_DIR]/cipd/bin/clang++", + "LDFLAGS": "--target=x86_64-apple-darwin --sysroot=/some/xcode/path" + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.mac-amd64.upload.upload binary to CAS", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@3@@@", + "@@@STEP_LINK@CAS UI@https://cas-viewer.appspot.com/projects/example-cas-server/instances/default_instance/blobs/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/0/tree@@@" + ] + }, + { "cmd": [ "xcrun", "--show-sdk-path" @@ -730,6 +909,51 @@ ] }, { + "cmd": [], + "name": "release.mac-arm64.upload", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@2@@@" + ] + }, + { + "cmd": [ + "[START_DIR]/cipd_tool/infra/tools/luci/cas/git_revision%3Amock_infra_git_revision/cas", + "archive", + "-cas-instance", + "projects/example-cas-server/instances/default_instance", + "-dump-digest", + "/path/to/tmp/", + "-paths", + "[START_DIR]/gn/out:gn" + ], + "cwd": "[START_DIR]/gn", + "env": { + "AR": "[START_DIR]/cipd/bin/llvm-ar", + "CC": "[START_DIR]/cipd/bin/clang", + "CFLAGS": "--target=arm64-apple-darwin --sysroot=/some/xcode/path -nostdinc++ -cxx-isystem [CACHE]/macos_sdk/XCode.app/include/c++/v1", + "CXX": "[START_DIR]/cipd/bin/clang++", + "LDFLAGS": "--target=arm64-apple-darwin --sysroot=/some/xcode/path" + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.mac-arm64.upload.upload binary to CAS", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@3@@@", + "@@@STEP_LINK@CAS UI@https://cas-viewer.appspot.com/projects/example-cas-server/instances/default_instance/blobs/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/0/tree@@@" + ] + }, + { "cmd": [ "sudo", "xcode-select",
diff --git a/infra/recipes/gn.expected/cq_win.json b/infra/recipes/gn.expected/cq_win.json index df7c728..e1f742b 100644 --- a/infra/recipes/gn.expected/cq_win.json +++ b/infra/recipes/gn.expected/cq_win.json
@@ -495,6 +495,189 @@ ] }, { + "cmd": [], + "name": "release.windows-amd64.upload", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@2@@@" + ] + }, + { + "cmd": [ + "vpython", + "-u", + "RECIPE_MODULE[recipe_engine::file]\\resources\\fileutil.py", + "--json-output", + "/path/to/tmp/json", + "copy", + "RECIPE_MODULE[recipe_engine::cas]\\resources\\infra.sha1", + "/path/to/tmp/" + ], + "cwd": "[START_DIR]\\gn", + "env": { + "VSINSTALLDIR": "[CACHE]\\windows_sdk" + }, + "env_prefixes": { + "PATH": [ + "[CACHE]\\windows_sdk\\win_sdk\\bin\\x64" + ] + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.windows-amd64.upload.read infra revision", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@3@@@", + "@@@STEP_LOG_LINE@infra.sha1@git_revision:mock_infra_git_revision@@@", + "@@@STEP_LOG_END@infra.sha1@@@" + ] + }, + { + "cmd": [], + "name": "release.windows-amd64.upload.install infra/tools/luci/cas", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@3@@@" + ] + }, + { + "cmd": [ + "vpython", + "-u", + "RECIPE_MODULE[recipe_engine::file]\\resources\\fileutil.py", + "--json-output", + "/path/to/tmp/json", + "ensure-directory", + "--mode", + "0777", + "[START_DIR]\\cipd_tool\\infra\\tools\\luci\\cas\\git_revision%3Amock_infra_git_revision" + ], + "cwd": "[START_DIR]\\gn", + "env": { + "VSINSTALLDIR": "[CACHE]\\windows_sdk" + }, + "env_prefixes": { + "PATH": [ + "[CACHE]\\windows_sdk\\win_sdk\\bin\\x64" + ] + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.windows-amd64.upload.install infra/tools/luci/cas.ensure package directory", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@4@@@" + ] + }, + { + "cmd": [ + "cipd.bat", + "ensure", + "-root", + "[START_DIR]\\cipd_tool\\infra\\tools\\luci\\cas\\git_revision%3Amock_infra_git_revision", + "-ensure-file", + "infra/tools/luci/cas/${platform} git_revision:mock_infra_git_revision", + "-max-threads", + "0", + "-json-output", + "/path/to/tmp/json" + ], + "cwd": "[START_DIR]\\gn", + "env": { + "VSINSTALLDIR": "[CACHE]\\windows_sdk" + }, + "env_prefixes": { + "PATH": [ + "[CACHE]\\windows_sdk\\win_sdk\\bin\\x64" + ] + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.windows-amd64.upload.install infra/tools/luci/cas.ensure_installed", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@4@@@", + "@@@STEP_LOG_LINE@json.output@{@@@", + "@@@STEP_LOG_LINE@json.output@ \"result\": {@@@", + "@@@STEP_LOG_LINE@json.output@ \"\": [@@@", + "@@@STEP_LOG_LINE@json.output@ {@@@", + "@@@STEP_LOG_LINE@json.output@ \"instance_id\": \"resolved-instance_id-of-git_revision:moc\", @@@", + "@@@STEP_LOG_LINE@json.output@ \"package\": \"infra/tools/luci/cas/resolved-platform\"@@@", + "@@@STEP_LOG_LINE@json.output@ }@@@", + "@@@STEP_LOG_LINE@json.output@ ]@@@", + "@@@STEP_LOG_LINE@json.output@ }@@@", + "@@@STEP_LOG_LINE@json.output@}@@@", + "@@@STEP_LOG_END@json.output@@@" + ] + }, + { + "cmd": [ + "[START_DIR]\\cipd_tool\\infra\\tools\\luci\\cas\\git_revision%3Amock_infra_git_revision\\cas", + "archive", + "-cas-instance", + "projects/example-cas-server/instances/default_instance", + "-dump-digest", + "/path/to/tmp/", + "-paths", + "[START_DIR]\\gn\\out:gn.exe" + ], + "cwd": "[START_DIR]\\gn", + "env": { + "VSINSTALLDIR": "[CACHE]\\windows_sdk" + }, + "env_prefixes": { + "PATH": [ + "[CACHE]\\windows_sdk\\win_sdk\\bin\\x64" + ] + }, + "infra_step": true, + "luci_context": { + "realm": { + "name": "gn:try" + }, + "resultdb": { + "current_invocation": { + "name": "invocations/build:8945511751514863184", + "update_token": "token" + }, + "hostname": "rdbhost" + } + }, + "name": "release.windows-amd64.upload.upload binary to CAS", + "~followup_annotations": [ + "@@@STEP_NEST_LEVEL@3@@@", + "@@@STEP_LINK@CAS UI@https://cas-viewer.appspot.com/projects/example-cas-server/instances/default_instance/blobs/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/0/tree@@@" + ] + }, + { "cmd": [ "taskkill.exe", "/f",
diff --git a/infra/recipes/gn.py b/infra/recipes/gn.py index bfbc4f7..0f8d470 100644 --- a/infra/recipes/gn.py +++ b/infra/recipes/gn.py
@@ -9,6 +9,7 @@ DEPS = [ 'recipe_engine/buildbucket', + 'recipe_engine/cas', 'recipe_engine/cipd', 'recipe_engine/context', 'recipe_engine/file', @@ -260,16 +261,20 @@ if target.is_host: api.step('test', [src_dir.join('out', 'gn_unittests')]) - if build_input.gerrit_changes: - continue - if config['name'] != 'release': continue with api.step.nest('upload'): - cipd_pkg_name = 'gn/gn/%s' % target.platform gn = 'gn' + ('.exe' if target.is_win else '') + if build_input.gerrit_changes: + # Upload to CAS from CQ. + api.cas.archive('upload binary to CAS', src_dir.join('out'), + src_dir.join('out', gn)) + continue + + cipd_pkg_name = 'gn/gn/%s' % target.platform + pkg_def = api.cipd.PackageDefinition( package_name=cipd_pkg_name, package_root=src_dir.join('out'),