| commit | 6ed85ec5155d9933b44a4aa5e132c9833090a6ca | [log] [tgz] |
|---|---|---|
| author | David 'Digit' Turner <digit@google.com> | Fri Apr 03 13:16:46 2020 +0200 |
| committer | Commit Bot <commit-bot@chromium.org> | Tue May 05 12:38:59 2020 +0000 |
| tree | 2f02d4b0ed7da9601c9f8753480c309b391b3b01 | |
| parent | 7c5d1002f4de08ca336a1bfee12272f875a60a66 [diff] |
Speedup --json=ide and --export-compile-commands=default
This CL significantly speeds up the generation of the project.json,
as well as that of compiled_commands.json. Note that these files
can be very large (e.g. 74 MiB / 6.4 MiB for Chromium, and
139 MiB / 45 MiB for Fuchsia), and are required by every Fuchsia
'fx gen' invocation.
Measurements [1] shows 2 seconds saved in the Fuchsia case, and
0.5s in the Chromium one. More importantly, this reduces peak
RSS usage by 323 MiB on Chromium, and 515 MiB on Fuchsia.
- compile_commands_writer.cc: output content directly to an
std::ostream instead of using an intermediate std::string,
using a StringBufferOutput for storage / file comparison /
file write.
- json_project_writer.cc: also use an std::ostream instead of
using an intermediate std::string, as well as implement its
own simple JSON writer class that doesn't require building
a complete in-memory representation of the whole document
before converting it into a string.
- rust_project_writer.cc: use StringOutputBuffer to avoid
touching the file if the content didn't change. Also
provide a minimal speedup (e.g. 90ms -> 70ms).
[1] Measurements are performed against the ToT GN (gn-master)
and the optimized one that results from this CL and 2
preparatory parent CLs (gn-json-writer).
Both are built with --use-icf --use-lto with the same
recent Clang toolchain. Tests are run on a Linux
workstation.
//
// CHROMIUM BEFORE
//
// NOTE: Command outputs were-reordered for better readability.
//
/work/chromium0/src$ repeat_cmd 5 /tmp/gn-master gen --ide=json
--export-compile-commands=default out/Release
Generating JSON projects took 1989ms (best)
Generating JSON projects took 2146ms
Generating JSON projects took 1993ms
Generating JSON projects took 2017ms
Generating JSON projects took 2167ms
Generating compile_commands took 45ms
Generating compile_commands took 61ms
Generating compile_commands took 41ms (best)
Generating compile_commands took 70ms
Generating compile_commands took 71ms
Done. Made 13413 targets from 2220 files in 7476ms
Done. Made 13413 targets from 2220 files in 7140ms
Done. Made 13413 targets from 2220 files in 6780ms (best)
Done. Made 13413 targets from 2220 files in 7022ms
Done. Made 13413 targets from 2220 files in 7203ms
//
// CHROMIUM AFTER
//
// 0.5s faster, 323 MiB less RAM.
//
/work/chromium0/src$ repeat_cmd 5 /tmp/gn-json-writer gen --ide=json
--export-compile-commands=default out/Release
Generating JSON projects took 1442ms
Generating JSON projects took 1523ms
Generating JSON projects took 1422ms (best)
Generating JSON projects took 1591ms
Generating JSON projects took 1579ms
Generating compile_commands took 24ms
Generating compile_commands took 25ms
Generating compile_commands took 23ms (best)
Generating compile_commands took 35ms
Generating compile_commands took 45ms
Done. Made 13413 targets from 2220 files in 6284ms
Done. Made 13413 targets from 2220 files in 6425ms
Done. Made 13413 targets from 2220 files in 6265ms (best)
Done. Made 13413 targets from 2220 files in 6434ms
Done. Made 13413 targets from 2220 files in 6541ms
/work/chromium0/src$ /usr/bin/time -f %M /tmp/gn-master gen --ide=json
--export-compile-commands=default out/Release
...
910784
/work/chromium0/src$ /usr/bin/time -f %M /tmp/gn-json-writer gen
--ide=json --export-compile-commands=default out/Release
...
579244
//
// FUCHSIA BEFORE
//
/work/fx-gn$ cp -f /tmp/gn-master prebuilt/third_party/gn/linux-x64/gn
/work/fx-gn$ repeat_cmd 5 prebuilt/third_party/gn/linux-x64/gn gen --ide=json --export-compile-commands=default out/default
Generating JSON projects took 4979ms
Generating JSON projects took 4860ms
Generating JSON projects took 5603ms
Generating JSON projects took 4683ms (best)
Generating JSON projects took 5346ms
Generating compile_commands took 655ms
Generating compile_commands took 589ms (best)
Generating compile_commands took 701ms
Generating compile_commands took 601ms
Generating compile_commands took 668ms
Done. Made 41018 targets from 3754 files in 18888ms
Done. Made 41018 targets from 3754 files in 18969ms
Done. Made 41018 targets from 3754 files in 20393ms
Done. Made 41018 targets from 3754 files in 18316ms (best)
Done. Made 41018 targets from 3754 files in 19534ms
//
// FUCHSIA AFTER
//
// 2s faster, 505 MiB less RAM.
//
/work/fx-gn$ cp -f /tmp/gn-json-writer prebuilt/third_party/gn/linux-x64/gn
/work/fx-gn$ repeat_cmd 5 prebuilt/third_party/gn/linux-x64/gn gen --ide=json --export-compile-commands=default out/default
Generating JSON projects took 2835ms (best)
Generating JSON projects took 2893ms
Generating JSON projects took 2877ms
Generating JSON projects took 3211ms
Generating JSON projects took 3073ms
Generating compile_commands took 399ms (best)
Generating compile_commands took 426ms
Generating compile_commands took 415ms
Generating compile_commands took 443ms
Generating compile_commands took 438ms
Done. Made 41018 targets from 3754 files in 16504ms
Done. Made 41018 targets from 3754 files in 16323ms (best)
Done. Made 41018 targets from 3754 files in 16669ms
Done. Made 41018 targets from 3754 files in 17366ms
Done. Made 41018 targets from 3754 files in 16698ms
/work/fx-gn$ /usr/bin/time -f %M prebuilt/third_party/gn/linux-x64/gn gen --ide=json --export-compile-commands=default out/default
...
1546356
/work/fx-gn$ /usr/bin/time -f %M prebuilt/third_party/gn/linux-x64/gn gen --ide=json --export-compile-commands=default out/default
...
1028656
Change-Id: Ie1a4dcd11f0798ca70b42f67f76a37bd2ea5c94a
Reviewed-on: https://gn-review.googlesource.com/c/gn/+/7942
Commit-Queue: David Turner <digit@google.com>
Reviewed-by: Scott Graham <scottmg@chromium.org>
GN is a meta-build system that generates build files for Ninja.
Related resources:
You can download the latest version of GN binary for Linux, macOS and Windows.
Alternatively, you can build GN from source:
git clone https://gn.googlesource.com/gn cd gn python build/gen.py ninja -C out # To run tests: out/gn_unittests
On Windows, it is expected that cl.exe, link.exe, and lib.exe can be found in PATH, so you'll want to run from a Visual Studio command prompt, or similar.
On Linux and Mac, the default compiler is clang++, a recent version is expected to be found in PATH. This can be overridden by setting CC, CXX, and AR.
There is a simple example in examples/simple_build directory that is a good place to get started with the minimal configuration.
To build and run the simple example with the default gcc compiler:
cd examples/simple_build ../../out/gn gen -C out ninja -C out ./out/hello
For a maximal configuration see the Chromium setup:
and the Fuchsia setup:
If you find a bug, you can see if it is known or report it in the bug database.
GN uses Gerrit for code review. The short version of how to patch is:
Register at https://gn-review.googlesource.com. ... edit code ... ninja -C out && out/gn_unittests
Then, to upload a change for review:
git commit git push origin HEAD:refs/for/master
The first time you do this you'll get an error from the server about a missing change-ID. Follow the directions in the error message to install the change-ID hook and run git commit --amend to apply the hook to the current commit.
When revising a change, use:
git commit --amend git push origin HEAD:refs/for/master
which will add the new changes to the existing code review, rather than creating a new one.
We ask that all contributors sign Google's Contributor License Agreement (either individual or corporate as appropriate, select ‘any other Google project’).
You may ask questions and follow along with GN‘s development on Chromium’s gn-dev@ Google Group.