Update and add metadata documentation Change-Id: Ic3cadb3ad5523301245ad8565a1bc9a12b79b02d Reviewed-on: https://gn-review.googlesource.com/c/gn/+/5640 Reviewed-by: Brett Wilson <brettw@chromium.org> Commit-Queue: Julie Hockett <juliehockett@google.com>
diff --git a/docs/reference.md b/docs/reference.md index 3a98061..1601d01 100644 --- a/docs/reference.md +++ b/docs/reference.md
@@ -154,6 +154,7 @@ * [input_conversion: Processing input from exec_script and read_file.](#io_conversion) * [label_pattern: Matching more than one label.](#label_pattern) * [labels: About labels.](#labels) + * [metadata_collection: About metadata and its collection.](#metadata_collection) * [ninja_rules: How Ninja build rules are named.](#ninja_rules) * [nogncheck: Annotating includes for checking.](#nogncheck) * [output_conversion: Specifies how to transform a value to output.](#io_conversion) @@ -6079,13 +6080,13 @@ ### <a name="var_walk_keys"></a>**walk_keys**: Key(s) for managing the metadata collection walk. ``` - Defaults to []. + Defaults to [""]. These keys are used to control the next step in a collection walk, acting as barriers. If a specified key is defined in a target's metadata, the walk will use the targets listed in that value to determine which targets are walked. - If no walk_keys are specified for a generated_file target (i.e. "[]"), the + If no walk_keys are specified for a generated_file target (i.e. "[""]"), the walk will touch all deps and data_deps of the specified target recursively. See "gn help generated_file". @@ -6822,6 +6823,147 @@ //net -> //net:net //tools/gn -> //tools/gn:gn ``` +### <a name="metadata_collection"></a>**Metadata Collection** + +``` + Metadata is information attached to targets throughout the dependency tree. GN + allows for the collection of this data into files written during the generation + step, enabing users to expose and aggregate this data based on the dependency + tree. +``` + +#### **generated_file targets** + +``` + Similar to the write_file() function, the generated_file target type + creates a file in the specified location with the specified content. The + primary difference between the function and the target type is that the + write_file function does the file write at parse time, while the + generated_file target type writes at target resolution time. See + "gn help generated_file" for more detail. + + When written at target resolution time, the generated_file enables GN to + collect and write aggregated metadata from dependents. + + A generated_file target can declare either 'contents' (to write statically + known contents to a file) or 'data_keys'(to aggregate metadata and write the + result to a file). It can also specify 'walk_keys' (to restrict the metadata + collection), 'output_conversion', and 'rebase'. +``` + +#### **Collection and Aggregation** + +``` + Targets can declare a 'metadata' variable containing a scope, and this + metadata is collected and written to file by generated_file aggregation + targets. The 'metadata' scope must contain only list values, since the + aggregation step collects a list of these values. + + During the target resolution, generated_file targets will walk their + dependencies recursively, collecting metadata based on the specified + 'data_keys'. 'data_keys' is specified as a list of strings, used by the walk + to identify which variables in dependencies' 'metadata' scopes to collect. + + The walk begins with the listed dependencies of the 'generated_file' target, + for each checking the "metadata" scope for any of the "data_keys". If + present, the data in those variables is appended to the aggregate list. Note + that this means that if more than one walk key is specified, the data in all + of them will be aggregated into one list. From there, the walk will then + recurse into the dependencies of each target it encounters, collecting the + specified metadata for each. + + For example: + + group("a") { + metadata = { + doom_melon = [ "enable" ] + my_files = [ "foo.cpp" ] + my_extra_files = [ "bar.cpp" ] + } + + deps = [ ":b" ] + } + + group("b") { + metadata = { + my_files = [ "baz.cpp" ] + } + } + + generated_file("metadata") { + outputs = [ "$root_build_dir/my_files.json" ] + data_keys = [ "my_files", "my_extra_files" ] + + deps = [ ":a" ] + } + + The above will produce the following file data: + + foo.cpp + bar.cpp + baz.cpp + + The dependency walk can be limited by using the "walk_keys". This is a list of + labels that should be included in the walk. All labels specified here should + also be in one of the deps lists. These keys act as barriers, where the walk + will only recurse into targets listed here. An empty list in all specified + barriers will end that portion of the walk. + + group("a") { + metadata = { + my_files = [ "foo.cpp" ] + my_files_barrier [ ":b" ] + } + + deps = [ ":b", ":c" ] + } + + group("b") { + metadata = { + my_files = [ "bar.cpp" ] + } + } + + group("c") { + metadata = { + my_files = [ "doom_melon.cpp" ] + } + } + + generated_file("metadata") { + outputs = [ "$root_build_dir/my_files.json" ] + data_keys = [ "my_files", "my_extra_files" ] + + deps = [ ":a" ] + } + + The above will produce the following file data (note that `doom_melon.cpp` is + not included): + + foo.cpp + bar.cpp + + A common example of this sort of barrier is in builds that have host tools + built as part of the tree, but do not want the metadata from those host tools + to be collected with the target-side code. +``` + +#### **Common Uses** + +``` + Metadata can be used to collect information about the different targets in the + build, and so a common use is to provide post-build tooling with a set of data + necessary to do aggregation tasks. For example, if each test target specifies + the output location of its binary to run in a metadata field, that can be + collected into a single file listing the locations of all tests in the + dependency tree. A local build tool (or continuous integration infrastructure) + can then use that file to know which tests exist, and where, and run them + accordingly. + + Another use is in image creation, where a post-build image tool needs to know + various pieces of information about the components it should include in order + to put together the correct image. +``` ### <a name="ninja_rules"></a>**Ninja build rules** #### **The "all" and "default" rules**
diff --git a/tools/gn/command_help.cc b/tools/gn/command_help.cc index 78c9161..7145da4 100644 --- a/tools/gn/command_help.cc +++ b/tools/gn/command_help.cc
@@ -13,6 +13,7 @@ #include "tools/gn/input_conversion.h" #include "tools/gn/label.h" #include "tools/gn/label_pattern.h" +#include "tools/gn/metadata.h" #include "tools/gn/ninja_build_writer.h" #include "tools/gn/output_conversion.h" #include "tools/gn/parser.h" @@ -83,6 +84,8 @@ PrintShortHelp("label_pattern: Matching more than one label.", "label_pattern"); PrintShortHelp("labels: About labels.", "labels"); + PrintShortHelp("metadata_collection: About metadata and its collection.", + "metadata_collection"); PrintShortHelp("ninja_rules: How Ninja build rules are named.", "ninja_rules"); PrintShortHelp("nogncheck: Annotating includes for checking.", "nogncheck"); @@ -93,7 +96,8 @@ "runtime_deps"); PrintShortHelp("source_expansion: Map sources to outputs for scripts.", "source_expansion"); - PrintShortHelp("switches: Show available command-line switches.", "switch_list"); + PrintShortHelp("switches: Show available command-line switches.", + "switch_list"); } void PrintSwitchHelp() { @@ -108,7 +112,8 @@ Do "gn help --the_switch_you_want_help_on" for more. Individual commands may take command-specific switches not listed here. See the help on your specific command for more. -)", "switch_list"); +)", + "switch_list"); if (is_markdown) OutputString("```\n", DECORATION_NONE); @@ -195,6 +200,7 @@ PrintLongHelp(kInputOutputConversion_Help, "io_conversion"); PrintLongHelp(kLabelPattern_Help, "label_pattern"); PrintLongHelp(kLabels_Help, "labels"); + PrintLongHelp(kMetadata_Help, "metadata_collection"); PrintLongHelp(kNinjaRules_Help, "ninja_rules"); PrintLongHelp(kNoGnCheck_Help, "nogncheck"); PrintLongHelp(kRuntimeDeps_Help, "runtime_deps"); @@ -335,6 +341,9 @@ }; random_topics["label_pattern"] = []() { PrintLongHelp(kLabelPattern_Help); }; random_topics["labels"] = []() { PrintLongHelp(kLabels_Help); }; + random_topics["metadata_collection"] = []() { + PrintLongHelp(kMetadata_Help); + }; random_topics["ninja_rules"] = []() { PrintLongHelp(kNinjaRules_Help); }; random_topics["nogncheck"] = []() { PrintLongHelp(kNoGnCheck_Help); }; random_topics["runtime_deps"] = []() { PrintLongHelp(kRuntimeDeps_Help); };
diff --git a/tools/gn/metadata.cc b/tools/gn/metadata.cc index 5091645..bc4c272 100644 --- a/tools/gn/metadata.cc +++ b/tools/gn/metadata.cc
@@ -6,6 +6,143 @@ #include "tools/gn/filesystem_utils.h" +const char kMetadata_Help[] = + R"(Metadata Collection + + Metadata is information attached to targets throughout the dependency tree. GN + allows for the collection of this data into files written during the generation + step, enabing users to expose and aggregate this data based on the dependency + tree. + +generated_file targets + + Similar to the write_file() function, the generated_file target type + creates a file in the specified location with the specified content. The + primary difference between the function and the target type is that the + write_file function does the file write at parse time, while the + generated_file target type writes at target resolution time. See + "gn help generated_file" for more detail. + + When written at target resolution time, the generated_file enables GN to + collect and write aggregated metadata from dependents. + + A generated_file target can declare either 'contents' (to write statically + known contents to a file) or 'data_keys'(to aggregate metadata and write the + result to a file). It can also specify 'walk_keys' (to restrict the metadata + collection), 'output_conversion', and 'rebase'. + + +Collection and Aggregation + + Targets can declare a 'metadata' variable containing a scope, and this + metadata is collected and written to file by generated_file aggregation + targets. The 'metadata' scope must contain only list values, since the + aggregation step collects a list of these values. + + During the target resolution, generated_file targets will walk their + dependencies recursively, collecting metadata based on the specified + 'data_keys'. 'data_keys' is specified as a list of strings, used by the walk + to identify which variables in dependencies' 'metadata' scopes to collect. + + The walk begins with the listed dependencies of the 'generated_file' target, + for each checking the "metadata" scope for any of the "data_keys". If + present, the data in those variables is appended to the aggregate list. Note + that this means that if more than one walk key is specified, the data in all + of them will be aggregated into one list. From there, the walk will then + recurse into the dependencies of each target it encounters, collecting the + specified metadata for each. + + For example: + + group("a") { + metadata = { + doom_melon = [ "enable" ] + my_files = [ "foo.cpp" ] + my_extra_files = [ "bar.cpp" ] + } + + deps = [ ":b" ] + } + + group("b") { + metadata = { + my_files = [ "baz.cpp" ] + } + } + + generated_file("metadata") { + outputs = [ "$root_build_dir/my_files.json" ] + data_keys = [ "my_files", "my_extra_files" ] + + deps = [ ":a" ] + } + + The above will produce the following file data: + + foo.cpp + bar.cpp + baz.cpp + + The dependency walk can be limited by using the "walk_keys". This is a list of + labels that should be included in the walk. All labels specified here should + also be in one of the deps lists. These keys act as barriers, where the walk + will only recurse into targets listed here. An empty list in all specified + barriers will end that portion of the walk. + + group("a") { + metadata = { + my_files = [ "foo.cpp" ] + my_files_barrier [ ":b" ] + } + + deps = [ ":b", ":c" ] + } + + group("b") { + metadata = { + my_files = [ "bar.cpp" ] + } + } + + group("c") { + metadata = { + my_files = [ "doom_melon.cpp" ] + } + } + + generated_file("metadata") { + outputs = [ "$root_build_dir/my_files.json" ] + data_keys = [ "my_files", "my_extra_files" ] + + deps = [ ":a" ] + } + + The above will produce the following file data (note that `doom_melon.cpp` is + not included): + + foo.cpp + bar.cpp + + A common example of this sort of barrier is in builds that have host tools + built as part of the tree, but do not want the metadata from those host tools + to be collected with the target-side code. + +Common Uses + + Metadata can be used to collect information about the different targets in the + build, and so a common use is to provide post-build tooling with a set of data + necessary to do aggregation tasks. For example, if each test target specifies + the output location of its binary to run in a metadata field, that can be + collected into a single file listing the locations of all tests in the + dependency tree. A local build tool (or continuous integration infrastructure) + can then use that file to know which tests exist, and where, and run them + accordingly. + + Another use is in image creation, where a post-build image tool needs to know + various pieces of information about the components it should include in order + to put together the correct image. +)"; + bool Metadata::WalkStep(const BuildSettings* settings, const std::vector<std::string>& keys_to_extract, const std::vector<std::string>& keys_to_walk,
diff --git a/tools/gn/metadata.h b/tools/gn/metadata.h index 06bd495..7b6168a 100644 --- a/tools/gn/metadata.h +++ b/tools/gn/metadata.h
@@ -11,6 +11,8 @@ #include "tools/gn/scope.h" #include "tools/gn/source_dir.h" +extern const char kMetadata_Help[]; + // Metadata about a particular target. // // Metadata is a collection of keys and values relating to a particular target.
diff --git a/tools/gn/variables.cc b/tools/gn/variables.cc index b35443d..74781aa 100644 --- a/tools/gn/variables.cc +++ b/tools/gn/variables.cc
@@ -2042,13 +2042,13 @@ const char kWalkKeys_Help[] = R"(walk_keys: Key(s) for managing the metadata collection walk. - Defaults to []. + Defaults to [""]. These keys are used to control the next step in a collection walk, acting as barriers. If a specified key is defined in a target's metadata, the walk will use the targets listed in that value to determine which targets are walked. - If no walk_keys are specified for a generated_file target (i.e. "[]"), the + If no walk_keys are specified for a generated_file target (i.e. "[""]"), the walk will touch all deps and data_deps of the specified target recursively. See "gn help generated_file".