Add clean_stale command
This CL adds a utility command to gn for cleaning out stale data from
build directories. As users change configurations, arguments or gn
files, the ninja build graph gets updated, but files and ninja database
entries remain. This command is effectively a wrapper around a pair of
ninja tools: cleandead and recompact. The former removes no longer
needed output files from the build directory and the latter prunes their
entries from the .ninja_log and .ninja_deps files.
Additionally, --clean-stale has been added as an option to the gen
command.
Change-Id: Ic931c1ac9fdf954d9a40318c7747ac9992a64045
Reviewed-on: https://gn-review.googlesource.com/c/gn/+/10401
Commit-Queue: RJ Ascani <rjascani@google.com>
Reviewed-by: Brett Wilson <brettw@chromium.org>
diff --git a/build/gen.py b/build/gen.py
index 0f40b75..922bf46 100755
--- a/build/gen.py
+++ b/build/gen.py
@@ -511,6 +511,7 @@
'src/gn/command_args.cc',
'src/gn/command_check.cc',
'src/gn/command_clean.cc',
+ 'src/gn/command_clean_stale.cc',
'src/gn/command_desc.cc',
'src/gn/command_format.cc',
'src/gn/command_gen.cc',
diff --git a/docs/reference.md b/docs/reference.md
index 8eb1510..4cced58 100644
--- a/docs/reference.md
+++ b/docs/reference.md
@@ -9,6 +9,7 @@
* [args: Display or configure arguments declared by the build.](#cmd_args)
* [check: Check header dependencies.](#cmd_check)
* [clean: Cleans the output directory.](#cmd_clean)
+ * [clean_stale: Cleans the stale output files from the output directory.](#cmd_clean_stale)
* [desc: Show lots of insightful information about a target or config.](#cmd_desc)
* [format: Format .gn files.](#cmd_format)
* [gen: Generate ninja files.](#cmd_gen)
@@ -486,6 +487,24 @@
Deletes the contents of the output directory except for args.gn and
creates a Ninja build environment sufficient to regenerate the build.
```
+### <a name="cmd_clean_stale"></a>**gn clean_stale [\--ninja-executable=...] <out_dir>...**
+
+```
+ Removes the no longer needed output files from the build directory and prunes
+ their records from the ninja build log and dependency database. These are
+ output files that were generated from previous builds, but the current build
+ graph no longer references them.
+
+ This command requires a ninja executable of at least version 1.10.0. The
+ executable must be provided by the --ninja-executable switch.
+```
+
+#### **Options**
+
+```
+ --ninja-executable=<string>
+ Can be used to specify the ninja executable to use.
+```
### <a name="cmd_desc"></a>**gn desc**
```
@@ -691,6 +710,12 @@
--stdin
Read input from stdin and write to stdout rather than update a file
in-place.
+
+ --read-tree=json
+ Reads an AST from stdin in the format output by --dump-tree=json and
+ uses that as the parse tree. (The only read-tree format currently
+ supported is json.) The given .gn file will be overwritten. This can be
+ used to programmatically transform .gn files.
```
#### **Examples**
@@ -699,6 +724,7 @@
gn format some\\BUILD.gn
gn format /abspath/some/BUILD.gn
gn format --stdin
+ gn format --read-tree=json //rewritten/BUILD.gn
```
### <a name="cmd_gen"></a>**gn gen [\--check] [<ide options>] <out_dir>**
@@ -725,7 +751,14 @@
Can be used to specify the ninja executable to use. This executable will
be used as an IDE option to indicate which ninja to use for building. This
executable will also be used as part of the gen process for triggering a
- restat on generated ninja files.
+ restat on generated ninja files and for use with --clean-stale.
+
+ --clean-stale
+ This option will cause no longer needed output files to be removed from
+ the build directory, and their records pruned from the ninja build log and
+ dependency database after the ninja build graph has been generated. This
+ option requires a ninja executable of at least version 1.10.0. It can be
+ provided by the --ninja-executable switch. Also see "gn help clean_stale".
```
#### **IDE options**
@@ -866,7 +899,6 @@
- "//foo:foo"
and not match:
- "//foo:bar"
-
```
### <a name="cmd_help"></a>**gn help <anything>**
diff --git a/src/gn/command_clean_stale.cc b/src/gn/command_clean_stale.cc
new file mode 100644
index 0000000..72f621d
--- /dev/null
+++ b/src/gn/command_clean_stale.cc
@@ -0,0 +1,98 @@
+// Copyright 2020 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#include <optional>
+
+#include "base/command_line.h"
+#include "base/files/file_path.h"
+#include "gn/commands.h"
+#include "gn/err.h"
+#include "gn/ninja_tools.h"
+#include "gn/setup.h"
+#include "gn/source_dir.h"
+#include "gn/switches.h"
+#include "gn/version.h"
+
+namespace commands {
+
+namespace {
+
+bool CleanStaleOneDir(const base::FilePath& ninja_executable,
+ const std::string& dir) {
+ // Deliberately leaked to avoid expensive process teardown.
+ Setup* setup = new Setup;
+ if (!setup->DoSetup(dir, false))
+ return false;
+
+ base::FilePath build_dir(setup->build_settings().GetFullPath(
+ SourceDir(setup->build_settings().build_dir().value())));
+
+ // The order of operations for these tools is:
+ // 1. cleandead - This eliminates old files from the build directory.
+ // 2. recompact - This prunes old entries from the ninja log and deps files.
+ //
+ // This order is ideal because the files removed by cleandead will no longer
+ // be found during the recompact, so ninja can prune their entries.
+ Err err;
+ if (!InvokeNinjaCleanDeadTool(ninja_executable, build_dir, &err)) {
+ err.PrintToStdout();
+ return false;
+ }
+
+ if (!InvokeNinjaRecompactTool(ninja_executable, build_dir, &err)) {
+ err.PrintToStdout();
+ return false;
+ }
+ return true;
+}
+
+} // namespace
+
+const char kCleanStale[] = "clean_stale";
+const char kCleanStale_HelpShort[] =
+ "clean_stale: Cleans the stale output files from the output directory.";
+const char kCleanStale_Help[] =
+ R"(gn clean_stale [--ninja-executable=...] <out_dir>...
+
+ Removes the no longer needed output files from the build directory and prunes
+ their records from the ninja build log and dependency database. These are
+ output files that were generated from previous builds, but the current build
+ graph no longer references them.
+
+ This command requires a ninja executable of at least version 1.10.0. The
+ executable must be provided by the --ninja-executable switch.
+
+Options
+
+ --ninja-executable=<string>
+ Can be used to specify the ninja executable to use.
+)";
+
+int RunCleanStale(const std::vector<std::string>& args) {
+ if (args.empty()) {
+ Err(Location(), "Missing argument.",
+ "Usage: \"gn clean_stale <out_dir>...\"")
+ .PrintToStdout();
+ return 1;
+ }
+
+ const base::CommandLine* cmdline = base::CommandLine::ForCurrentProcess();
+ base::FilePath ninja_executable = cmdline->GetSwitchValuePath(switches::kNinjaExecutable);
+ if (ninja_executable.empty()) {
+ Err(Location(), "No --ninja-executable provided.",
+ "--clean-stale requires a ninja executable to run. You can "
+ "provide one on the command line via --ninja-executable.")
+ .PrintToStdout();
+ return 1;
+ }
+
+ for (const std::string& dir : args) {
+ if (!CleanStaleOneDir(ninja_executable, dir))
+ return 1;
+ }
+
+ return 0;
+}
+
+} // namespace commands
diff --git a/src/gn/command_gen.cc b/src/gn/command_gen.cc
index 4437236..20bc332 100644
--- a/src/gn/command_gen.cc
+++ b/src/gn/command_gen.cc
@@ -33,6 +33,7 @@
namespace {
const char kSwitchCheck[] = "check";
+const char kSwitchCleanStale[] = "clean-stale";
const char kSwitchFilters[] = "filters";
const char kSwitchIde[] = "ide";
const char kSwitchIdeValueEclipse[] = "eclipse";
@@ -357,18 +358,43 @@
bool RunNinjaPostProcessTools(const BuildSettings* build_settings,
base::FilePath ninja_executable,
bool is_regeneration,
+ bool clean_stale,
Err* err) {
// If the user did not specify an executable, skip running the post processing
// tools. Since these tools can re-write ninja build log and dep logs, it is
// really important that ninja executable used for tools matches the
// executable that is used for builds.
if (ninja_executable.empty()) {
+ if (clean_stale) {
+ *err = Err(Location(), "No --ninja-executable provided.",
+ "--clean-stale requires a ninja executable to run. You can "
+ "provide one on the command line via --ninja-executable.");
+ return false;
+ }
+
return true;
}
base::FilePath build_dir =
build_settings->GetFullPath(build_settings->build_dir());
+ if (clean_stale) {
+ if (build_settings->ninja_required_version() < Version{1, 10, 0}) {
+ *err = Err(Location(), "Need a ninja executable at least version 1.10.0.",
+ "--clean-stale requires a ninja executable of version 1.10.0 "
+ "or later.");
+ return false;
+ }
+
+ if (!InvokeNinjaCleanDeadTool(ninja_executable, build_dir, err)) {
+ return false;
+ }
+
+ if(!InvokeNinjaRecompactTool(ninja_executable, build_dir, err)) {
+ return false;
+ }
+ }
+
// If we have a ninja version that supports restat, we should restat the
// build.ninja file so the next ninja invocation will use the right mtime. If
// gen is being invoked as part of a re-gen (ie, ninja is invoking gn gen),
@@ -413,7 +439,14 @@
Can be used to specify the ninja executable to use. This executable will
be used as an IDE option to indicate which ninja to use for building. This
executable will also be used as part of the gen process for triggering a
- restat on generated ninja files.
+ restat on generated ninja files and for use with --clean-stale.
+
+ --clean-stale
+ This option will cause no longer needed output files to be removed from
+ the build directory, and their records pruned from the ninja build log and
+ dependency database after the ninja build graph has been generated. This
+ option requires a ninja executable of at least version 1.10.0. It can be
+ provided by the --ninja-executable switch. Also see "gn help clean_stale".
IDE options
@@ -603,7 +636,8 @@
if (!RunNinjaPostProcessTools(
&setup->build_settings(),
command_line->GetSwitchValuePath(switches::kNinjaExecutable),
- command_line->HasSwitch(switches::kRegeneration), &err)) {
+ command_line->HasSwitch(switches::kRegeneration),
+ command_line->HasSwitch(kSwitchCleanStale), &err)) {
err.PrintToStdout();
return 1;
}
diff --git a/src/gn/commands.cc b/src/gn/commands.cc
index 97d2629..da6514b 100644
--- a/src/gn/commands.cc
+++ b/src/gn/commands.cc
@@ -438,6 +438,7 @@
INSERT_COMMAND(Outputs)
INSERT_COMMAND(Path)
INSERT_COMMAND(Refs)
+ INSERT_COMMAND(CleanStale);
#undef INSERT_COMMAND
}
diff --git a/src/gn/commands.h b/src/gn/commands.h
index 28d713a..54a1a9d 100644
--- a/src/gn/commands.h
+++ b/src/gn/commands.h
@@ -94,6 +94,11 @@
extern const char kRefs_Help[];
int RunRefs(const std::vector<std::string>& args);
+extern const char kCleanStale[];
+extern const char kCleanStale_HelpShort[];
+extern const char kCleanStale_Help[];
+int RunCleanStale(const std::vector<std::string>& args);
+
// -----------------------------------------------------------------------------
struct CommandInfo {
diff --git a/src/gn/ninja_tools.cc b/src/gn/ninja_tools.cc
index ed45544..2eaf31c 100644
--- a/src/gn/ninja_tools.cc
+++ b/src/gn/ninja_tools.cc
@@ -62,3 +62,21 @@
std::string output;
return RunNinja(cmdline, build_dir, &output, err);
}
+
+bool InvokeNinjaCleanDeadTool(const base::FilePath& ninja_executable,
+ const base::FilePath& build_dir,
+ Err* err) {
+ base::CommandLine cmdline =
+ CreateNinjaToolCommandLine(ninja_executable, "cleandead");
+ std::string output;
+ return RunNinja(cmdline, build_dir, &output, err);
+}
+
+bool InvokeNinjaRecompactTool(const base::FilePath& ninja_executable,
+ const base::FilePath& build_dir,
+ Err* err) {
+ base::CommandLine cmdline =
+ CreateNinjaToolCommandLine(ninja_executable, "recompact");
+ std::string output;
+ return RunNinja(cmdline, build_dir, &output, err);
+}
diff --git a/src/gn/ninja_tools.h b/src/gn/ninja_tools.h
index 4eda1e1..9721296 100644
--- a/src/gn/ninja_tools.h
+++ b/src/gn/ninja_tools.h
@@ -27,4 +27,18 @@
const std::vector<base::FilePath>& files_to_restat,
Err* err);
+// Invokes the ninja cleandead tool (ie, ninja -C build_dir -t cleandead). This
+// tool removes files produced by previous builds that are no longer in the
+// build file.
+bool InvokeNinjaCleanDeadTool(const base::FilePath& ninja_executable,
+ const base::FilePath& build_dir,
+ Err* err);
+
+// Invokes the ninja recompact tool (ie, ninja -C build_dir -t recompact). This
+// tool prunes the .ninja_log and .ninja_deps entries that are no longer part of
+// the build graph.
+bool InvokeNinjaRecompactTool(const base::FilePath& ninja_executable,
+ const base::FilePath& build_dir,
+ Err* err);
+
#endif // TOOLS_GN_NINJA_TOOLS_H_
