gn gen: Implement thread-local ResolvedTargetData cache.
Final benchmarks for 'gn gen' from the previous stack of CLs
shows significant time reduction for moderate peak RAM increase:
TIME_BEFORE TIME_AFTER RAM_BEFORE RAM_AFTER
Chromium: 4.75s 3.30s 1.00GiB 1.31GiB
Fuchsia: 20.99s 17.98s 3.84GiB 4.07GiB
Benchmark Details ------------------------------------------
Chromium:
$ hyperfine '/tmp/gn-main gen out/Default' '/tmp/gn-new gen out/Default'
Benchmark 1: /tmp/gn-main gen out/Default
Time (mean ± σ): 4.753 s ± 0.044 s [User: 15.413 s, System: 8.977 s]
Range (min … max): 4.672 s … 4.820 s 10 runs
Benchmark 2: /tmp/gn-new gen out/Default
Time (mean ± σ): 3.303 s ± 0.061 s [User: 18.202 s, System: 9.122 s]
Range (min … max): 3.217 s … 3.413 s 10 runs
Summary
'/tmp/gn-new gen out/Default' ran
1.44 ± 0.03 times faster than '/tmp/gn-main gen out/Default'
$ /usr/bin/time -f%M /tmp/gn-main gen out/Default
Done. Made 17871 targets from 3048 files in 4674ms
1049960
$ /usr/bin/time -f%M /tmp/gn-new gen out/Default
Done. Made 17871 targets from 3048 files in 3229ms
1374344
Fuchsia:
$ hyperfine '/tmp/gn-main gen out/default' '/tmp/gn-new gen out/default'
Benchmark 1: /tmp/gn-main gen out/default
Time (mean ± σ): 20.994 s ± 0.277 s [User: 97.113 s, System: 49.242 s]
Range (min … max): 20.637 s … 21.602 s 10 runs
Benchmark 2: /tmp/gn-new gen out/default
Time (mean ± σ): 17.981 s ± 0.259 s [User: 95.037 s, System: 48.343 s]
Range (min … max): 17.390 s … 18.324 s 10 runs
Summary
'/tmp/gn-new gen out/default' ran
1.17 ± 0.02 times faster than '/tmp/gn-main gen out/default'
$ /usr/bin/time -f%M /tmp/gn-main gen out/default
Generating compile_commands took 727ms
Done. Made 196612 targets from 6247 files in 20741ms
4026004
$ /usr/bin/time -f%M /tmp/gn-new gen out/default
Generating compile_commands took 659ms
Done. Made 196612 targets from 6247 files in 17427ms
4272248
Bug: 331
Change-Id: Ib2eca769a5a587a531f4fac2be263555e37b89ea
Reviewed-on: https://gn-review.googlesource.com/c/gn/+/15329
Commit-Queue: David Turner <digit@google.com>
Reviewed-by: Takuto Ikuta <tikuta@google.com>
diff --git a/src/gn/command_gen.cc b/src/gn/command_gen.cc
index 6216269..ec4a766 100644
--- a/src/gn/command_gen.cc
+++ b/src/gn/command_gen.cc
@@ -3,6 +3,8 @@
// found in the LICENSE file.
#include <mutex>
+#include <thread>
+#include <unordered_map>
#include "base/command_line.h"
#include "base/strings/string_number_conversions.h"
@@ -72,11 +74,23 @@
struct TargetWriteInfo {
std::mutex lock;
NinjaWriter::PerToolchainRules rules;
+
+ using ResolvedMap = std::unordered_map<std::thread::id, ResolvedTargetData>;
+ std::unique_ptr<ResolvedMap> resolved_map = std::make_unique<ResolvedMap>();
+
+ void LeakOnPurpose() {
+ (void)resolved_map.release();
+ }
};
// Called on worker thread to write the ninja file.
void BackgroundDoWrite(TargetWriteInfo* write_info, const Target* target) {
- std::string rule = NinjaTargetWriter::RunAndWriteFile(target);
+ ResolvedTargetData* resolved;
+ {
+ std::lock_guard<std::mutex> lock(write_info->lock);
+ resolved = &((*write_info->resolved_map)[std::this_thread::get_id()]);
+ }
+ std::string rule = NinjaTargetWriter::RunAndWriteFile(target, resolved);
DCHECK(!rule.empty());
{
@@ -787,6 +801,10 @@
OutputString(stats);
}
+ // Just like the build graph, leak the resolved data to avoid expensive
+ // process teardown here too.
+ write_info.LeakOnPurpose();
+
return 0;
}
