commit | 28044bc7e33163392d723b5d7bbe353a089108fb | [log] [tgz] |
---|---|---|
author | David 'Digit' Turner <digit@google.com> | Thu Nov 28 16:39:43 2019 +0100 |
committer | Commit Bot <commit-bot@chromium.org> | Tue Feb 04 23:40:44 2020 +0000 |
tree | b9576f5600507019f5e9db9c327773b67a12458e | |
parent | eb0c7518b080c93305d0060e91c2bca519cff6ed [diff] |
Speed up source file set operations. Profiling the Fuchsia Zircon build's "gn gen" operation [1] shows that a lot of time is spend copying source file sets during a very large number of scope merges. This CL provides a 30% speedup for the Zircon "gn gen", without degrading the Chromium one (see measurements below). This is done by introducing SourceFileSet, a dedicated data type with avoids std::string allocations, and has much faster lookup and insert / copy operations, compared with std::set<SourceFileSet> More precisely, by: - Introducing a StringAtom data type, modelling a pointer to a globally unique constant string, with a heavily optimized lookup implementation. See comments in src/gn/string_atom.h and src/gn/string_atom.cc for usage and full details. - Modifying SourceFile to use a StringAtom, instead of an std::string to store the file path. This reduces the size and copy cost of each instance. - Replace std::set<SourceFile> with the new SourceFileSet data type (defined as a base::flat_set<SourceFile>, with fast pointer-based comparisons, which turns out to be significantly faster than other alternatives). Note that this works well because source file paths are immutable and shared among many many scopes / SourceFileSet instances. Note that despite the fact that SourceFileSet is unordered for performance reasons, this CL does not modify the outputs of "gn gen", at least when used for a Chromium Linux build. See [3] for the methodology used to verify that. Finally note that the patch reduces memory usage by nearly 1 GiB (!!) for the Zircon build [4] (but only 6 MiB for the Chromium one). [1] Performed from a Fuchsia source workspace with: $ gn gen --root=zircon out/default.zircon [2] Results comparing the performance of 'gn' before and after the patch. Both binaries were built with a recent clang toolchain with build/gen.py -use-icf -use-lto: -------------------------------------------------------- // ZIRCON BUILD [fuchsia] $ gn gen --root=zircon out/default.zircon //// BEFORE (* = best of 5 runs) Done. Made 51431 targets from 944 files in 12825ms * Done. Made 51431 targets from 944 files in 13932ms Done. Made 51431 targets from 944 files in 14886ms Done. Made 51431 targets from 944 files in 14642ms Done. Made 51431 targets from 944 files in 13548ms //// AFTER (* = best of 5 runs) Done. Made 51431 targets from 944 files in 10623ms Done. Made 51431 targets from 944 files in 10185ms Done. Made 51431 targets from 944 files in 10623ms Done. Made 51431 targets from 944 files in 9759ms * Done. Made 51431 targets from 944 files in 10656ms Overall improvement: 12825 - 9759 = 3066 ms (31% faster) -------------------------------------------------------- // CHROMIUM BUILD [chromium/src] $ gn gen out/default //// BEFORE Done. Made 12150 targets from 2121 files in 4319ms Done. Made 12150 targets from 2121 files in 4545ms Done. Made 12150 targets from 2121 files in 4346ms Done. Made 12150 targets from 2121 files in 4298ms * Done. Made 12150 targets from 2121 files in 4390ms //// AFTER Done. Made 12150 targets from 2121 files in 4274ms Done. Made 12150 targets from 2121 files in 4223ms Done. Made 12150 targets from 2121 files in 4098ms Done. Made 12150 targets from 2121 files in 4419ms Done. Made 12150 targets from 2121 files in 4096ms * Overall improvement: 4298 - 4096 = 202 ms (4.9% faster) [3] To verify that "gn gen" outputs were not affected by this CL, the following experiment was setup on a Linux workstation. 1) Create a fresh chromium source checkout on a btrfs subvolume directory (e.g. /work/chromium, where /work is a btrfs mount point). 2) cd /work/chromium/src && touch out/args.gn # Create an empty args.gn there. 3) cd /work/chromium/src && <original-gn> gen out # Generate out/ files using original gn program. 4) cd /work && sudo btrfs subvolume snapshot chromium chromium-original # Snapshot the generated state to /work/chromium-original 4) cd /work/chromium/src && <new-gn> gen out # Regenerate out/ files using patched gn program. 5) diff -burN /work/chromium-original/src/out /work/chromium/src/out # Compare the files under out/ --- /work/chromium0/src/out/default/build.ninja 2020-01-31 13:24:32.643343218 +0100 +++ /work/cr-xx0/src/out/default/build.ninja 2020-01-31 13:21:42.105117112 +0100 @@ -1,7 +1,7 @@ ninja_required_version = 1.7.2 rule gn - command = ../../../../../tmp/gn-sourcefileset --root=../.. -q gen . + command = ../../../../../tmp/gn-master --root=../.. -q gen . description = Regenerating ninja files build build.ninja: gn The result shows that the only difference is the name of "gn" executable name captured in build.ninja, all other files being identical. [4] Peak resident set size (RSS) usage as reported with /usr/bin/time -v // ZIRCON BUILD BEFORE: Maximum resident set size (kbytes): 2793824 AFTER: Maximum resident set size (kbytes): 1817880 DIFF: 2793824 - 1817880 = 975944 kiB = 953 MiB (!!) // CHROMIUM BUILD BEFORE: Maximum resident set size (kbytes): 544296 AFTER: Maximum resident set size (kbytes): 538068 DIFF: 544296 - 538068 = 6228 kiB = 6.1 MiB Change-Id: I4f11fcddeb7b84a6126b748d2b4d66afd1642545 Reviewed-on: https://gn-review.googlesource.com/c/gn/+/7280 Commit-Queue: Brett Wilson <brettw@chromium.org> Reviewed-by: Brett Wilson <brettw@chromium.org> Reviewed-by: Scott Graham <scottmg@chromium.org>
GN is a meta-build system that generates build files for Ninja.
Related resources:
You can download the latest version of GN binary for Linux, macOS and Windows.
Alternatively, you can build GN from source:
git clone https://gn.googlesource.com/gn cd gn python build/gen.py ninja -C out # To run tests: out/gn_unittests
On Windows, it is expected that cl.exe
, link.exe
, and lib.exe
can be found in PATH
, so you'll want to run from a Visual Studio command prompt, or similar.
On Linux and Mac, the default compiler is clang++
, a recent version is expected to be found in PATH
. This can be overridden by setting CC
, CXX
, and AR
.
There is a simple example in examples/simple_build directory that is a good place to get started with the minimal configuration.
To build and run the simple example with the default gcc compiler:
cd examples/simple_build ../../out/gn gen -C out ninja -C out ./out/hello
For a maximal configuration see the Chromium setup:
and the Fuchsia setup:
If you find a bug, you can see if it is known or report it in the bug database.
GN uses Gerrit for code review. The short version of how to patch is:
Register at https://gn-review.googlesource.com. ... edit code ... ninja -C out && out/gn_unittests
Then, to upload a change for review:
git commit git push origin HEAD:refs/for/master
The first time you do this you'll get an error from the server about a missing change-ID. Follow the directions in the error message to install the change-ID hook and run git commit --amend
to apply the hook to the current commit.
When revising a change, use:
git commit --amend git push origin HEAD:refs/for/master
which will add the new changes to the existing code review, rather than creating a new one.
We ask that all contributors sign Google's Contributor License Agreement (either individual or corporate as appropriate, select ‘any other Google project’).
You may ask questions and follow along with GN‘s development on Chromium’s gn-dev@ Google Group.